2021-09-17T09:56:24.692 INFO:root:teuthology version: 1.1.0-4e1f59de 2021-09-17T09:56:24.694 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_prsrivas@teuthology --name prsrivas-2021-09-17_09:50:01-rgw-wip-rgw-sts-51219-distro-basic-smithi --archive /home/teuthworker/archive/prsrivas-2021-09-17_09:50:01-rgw-wip-rgw-sts-51219-distro-basic-smithi/6394893 --description rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{ubuntu_latest} tasks/test_multi} -- /home/teuthworker/archive/prsrivas-2021-09-17_09:50:01-rgw-wip-rgw-sts-51219-distro-basic-smithi/6394893/orig.config.yaml 2021-09-17T09:56:24.735 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-09-17T09:56:24.788 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/prsrivas-2021-09-17_09:50:01-rgw-wip-rgw-sts-51219-distro-basic-smithi/6394893 branch: wip-rgw-sts-51219 description: rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{ubuntu_latest} tasks/test_multi} email: null first_in_suite: false job_id: '6394893' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: prsrivas-2021-09-17_09:50:01-rgw-wip-rgw-sts-51219-distro-basic-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-rgw-sts-51219 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: f7edf77e75bbfd0d8d41a561025fbfc76479a1a3 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: f7edf77e75bbfd0d8d41a561025fbfc76479a1a3 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 rgw-multisite-tests: args: - tests.py - tests_ps.py workunit: branch: wip-rgw-sts-51219 sha1: f7edf77e75bbfd0d8d41a561025fbfc76479a1a3 owner: scheduled_prsrivas@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: f7edf77e75bbfd0d8d41a561025fbfc76479a1a3 sleep_before_teardown: 0 suite: rgw suite_branch: wip-rgw-sts-51219 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: f7edf77e75bbfd0d8d41a561025fbfc76479a1a3 targets: smithi018.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlJIDWGCVw0d6mjxlh5xHp6nUKmbLatg3/YS7e43l9KwOgism12PFx3Iiqtwxc+elL4poNbk7iEMx7XcCnD/QBrYN9C5W54VzlplpQHWXUE2cJhzQ7WG2gJ+oKOF4V6BTdapegDA/7kq8Jz3Rg+OfrZkTGJCU1JUFbCw1sIc8NyEPHGarNjmw9fk6mOI76kERNC2nJjycSGerLscjHddbF9pE2bh/gPMGObzEK/ejYi5bHZGIW4Lfo+12WuBY5bxXfnl1OwnR1sdUj1cQcPk4Pc0LTqRpV1NFq2A7ExZ82kPpVJqCyNWIBrdoaryV2DNxQjEAJxPKNxEDla5a48B85teYJlzGB5nsL+YLCzvJupUEGjHPuI8ekBh75Ytzf7dkZ2wKfbYkJ5jRKGZ35Obck3O6bQpchHvwtiljzmCARJEtWhYmLrR+eaLmpzFloDoXL/iUM28EUPjAWe9ivKDTJmuoi1vKE58mBzeM5XBCPThFwkUM+9JPUetPMPomvBOc= smithi050.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0ax+8bAaTUfgqqI1YjYyC3aVNh1vV6bqIxfjTx/he4L4dKnMe0z4v0rJ+gciLQfjseJ1m03CvuanS70IfSVHqbDPP82Ag6I0wqa4MLxEHO/gwfag/boLgNRPJHqq1IvxuRMDH3VeL8Y8tZZdBdPqtHUEaAUTOTqLsSaMUOvGVL17wR/+j9BtMeSFsk7hkL3EBepvGLNlRlSb96Z0IEkPOodAAv0iLWKRJMEZGSFnU6Uo4jnLmSX9mSVNuUuK5Z+ixi1n9EZ9SH1vxSjYVQ1dd2fTrEWLsyr0HLsOxcVy8QxTCuyM7zc0nAX2rWsHeVap5hDIbxcSxojIaTGhtF/54vXIgDwHTR88l24no1R1t4WVPKQ7rtpd6PafLjg8A91Dy+k7Xog4iyara0wGwCCIiLC1hQEjAPsjtfA4QMA2ckyUgMMX29xXQIIkcF/Ez54Aaim/UXYUxakohc8l+B3ZJOLDfFlX10KzjorA/xE3o9RbaA2p5nBvO+m3Auy9PBW0= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master teuthology_sha1: 4e1f59de82924f116bd6b91664f90ef632c60a3f timestamp: 2021-09-17_09:50:01 tube: smithi user: prsrivas verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.30581 2021-09-17T09:56:24.789 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/qa; will attempt to use it 2021-09-17T09:56:24.790 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/qa/tasks 2021-09-17T09:56:24.790 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-09-17T09:56:24.792 INFO:teuthology.task.internal:Checking packages... 2021-09-17T09:56:24.813 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3' 2021-09-17T09:56:24.813 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-09-17T09:56:24.814 INFO:teuthology.packaging:ref: None 2021-09-17T09:56:24.814 INFO:teuthology.packaging:tag: None 2021-09-17T09:56:24.814 INFO:teuthology.packaging:branch: wip-rgw-sts-51219 2021-09-17T09:56:24.815 INFO:teuthology.packaging:sha1: f7edf77e75bbfd0d8d41a561025fbfc76479a1a3 2021-09-17T09:56:24.815 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-rgw-sts-51219 2021-09-17T09:56:24.966 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-7741-gf7edf77e-1focal 2021-09-17T09:56:24.966 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-09-17T09:56:24.975 INFO:teuthology.task.internal:no buildpackages task found 2021-09-17T09:56:24.976 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-09-17T09:56:25.017 INFO:teuthology.task.internal:Saving configuration 2021-09-17T09:56:25.033 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-09-17T09:56:25.045 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-17T09:56:25.065 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-09-17 09:51:25.003761', 'description': '/home/teuthworker/archive/prsrivas-2021-09-17_09:50:01-rgw-wip-rgw-sts-51219-distro-basic-smithi/6394893', 'locked_by': 'scheduled_prsrivas@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDqcxIgy5AdH6vXAF4vi9v9Byz3BbnDXPmgsgU7dbFh6', 'os_version': '20.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi018.front.sepia.ceph.com'} 2021-09-17T09:56:25.082 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-09-17 09:51:25.003782', 'description': '/home/teuthworker/archive/prsrivas-2021-09-17_09:50:01-rgw-wip-rgw-sts-51219-distro-basic-smithi/6394893', 'locked_by': 'scheduled_prsrivas@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFcZDwg+fSO93msfAQ9le7vhpI1uMHptbD69w14gBrxr', 'os_version': '20.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi050.front.sepia.ceph.com'} 2021-09-17T09:56:25.082 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-09-17T09:56:25.095 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-09-17T09:56:25.095 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-09-17T09:56:25.095 INFO:teuthology.run_tasks:Running task console_log... 2021-09-17T09:56:25.167 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f252fdfaf28>, signals=[15]) 2021-09-17T09:56:25.168 INFO:teuthology.run_tasks:Running task internal.connect... 2021-09-17T09:56:25.179 INFO:teuthology.task.internal:Opening connections... 2021-09-17T09:56:25.179 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi018.front.sepia.ceph.com 2021-09-17T09:56:25.181 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-17T09:56:25.232 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2021-09-17T09:56:25.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-17T09:56:25.287 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-09-17T09:56:25.299 DEBUG:teuthology.orchestra.run.smithi018:> uname -m 2021-09-17T09:56:25.313 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64 2021-09-17T09:56:25.313 DEBUG:teuthology.orchestra.run.smithi018:> cat /etc/os-release 2021-09-17T09:56:25.361 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="Ubuntu" 2021-09-17T09:56:25.362 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-09-17T09:56:25.362 INFO:teuthology.orchestra.run.smithi018.stdout:ID=ubuntu 2021-09-17T09:56:25.362 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE=debian 2021-09-17T09:56:25.362 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-09-17T09:56:25.363 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="20.04" 2021-09-17T09:56:25.363 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://www.ubuntu.com/" 2021-09-17T09:56:25.363 INFO:teuthology.orchestra.run.smithi018.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-09-17T09:56:25.363 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-09-17T09:56:25.364 INFO:teuthology.orchestra.run.smithi018.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-09-17T09:56:25.364 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_CODENAME=focal 2021-09-17T09:56:25.364 INFO:teuthology.orchestra.run.smithi018.stdout:UBUNTU_CODENAME=focal 2021-09-17T09:56:25.365 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2021-09-17T09:56:25.381 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2021-09-17T09:56:25.388 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2021-09-17T09:56:25.388 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2021-09-17T09:56:25.434 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="Ubuntu" 2021-09-17T09:56:25.536 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-09-17T09:56:25.536 INFO:teuthology.orchestra.run.smithi050.stdout:ID=ubuntu 2021-09-17T09:56:25.537 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE=debian 2021-09-17T09:56:25.537 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-09-17T09:56:25.537 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="20.04" 2021-09-17T09:56:25.537 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://www.ubuntu.com/" 2021-09-17T09:56:25.538 INFO:teuthology.orchestra.run.smithi050.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-09-17T09:56:25.538 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-09-17T09:56:25.538 INFO:teuthology.orchestra.run.smithi050.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-09-17T09:56:25.538 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_CODENAME=focal 2021-09-17T09:56:25.539 INFO:teuthology.orchestra.run.smithi050.stdout:UBUNTU_CODENAME=focal 2021-09-17T09:56:25.539 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2021-09-17T09:56:25.553 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-09-17T09:56:25.619 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-09-17T09:56:25.629 INFO:teuthology.task.internal:Checking for old test directory... 2021-09-17T09:56:25.630 DEBUG:teuthology.orchestra.run.smithi018:> test '!' -e /home/ubuntu/cephtest 2021-09-17T09:56:25.632 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2021-09-17T09:56:25.638 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-09-17T09:56:25.651 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-09-17T09:56:25.651 DEBUG:teuthology.orchestra.run.smithi018:> test -z $(ls -A /var/lib/ceph) 2021-09-17T09:56:25.681 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2021-09-17T09:56:25.706 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-09-17T09:56:25.752 INFO:teuthology.run_tasks:Running task kernel... 2021-09-17T09:56:25.770 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-09-17T09:56:25.771 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-09-17T09:56:25.771 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2021-09-17T09:56:25.778 INFO:teuthology.orchestra.run.smithi018.stdout:5.4.0-66-generic 2021-09-17T09:56:25.778 INFO:teuthology.task.kernel:Running kernel on smithi018: 5.4.0-66-generic 2021-09-17T09:56:25.779 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get clean 2021-09-17T09:56:25.863 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get update 2021-09-17T09:56:26.009 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2021-09-17T09:56:26.127 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-09-17T09:56:26.210 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2021-09-17T09:56:26.292 INFO:teuthology.orchestra.run.smithi018.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [282 kB] 2021-09-17T09:56:26.352 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [867 kB] 2021-09-17T09:56:26.384 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [167 kB] 2021-09-17T09:56:26.387 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [8,712 B] 2021-09-17T09:56:26.387 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [406 kB] 2021-09-17T09:56:26.393 INFO:teuthology.orchestra.run.smithi018.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [18.0 kB] 2021-09-17T09:56:26.394 INFO:teuthology.orchestra.run.smithi018.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [58.3 kB] 2021-09-17T09:56:26.395 INFO:teuthology.orchestra.run.smithi018.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [508 B] 2021-09-17T09:56:26.395 INFO:teuthology.orchestra.run.smithi018.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [507 kB] 2021-09-17T09:56:26.401 INFO:teuthology.orchestra.run.smithi018.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [639 kB] 2021-09-17T09:56:26.408 INFO:teuthology.orchestra.run.smithi018.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [102 kB] 2021-09-17T09:56:26.409 INFO:teuthology.orchestra.run.smithi018.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [12.4 kB] 2021-09-17T09:56:26.422 INFO:teuthology.orchestra.run.smithi018.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [21.9 kB] 2021-09-17T09:56:26.422 INFO:teuthology.orchestra.run.smithi018.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [4,948 B] 2021-09-17T09:56:26.423 INFO:teuthology.orchestra.run.smithi018.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2021-09-17T09:56:26.556 INFO:teuthology.orchestra.run.smithi018.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB] 2021-09-17T09:56:26.698 INFO:teuthology.orchestra.run.smithi018.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [536 kB] 2021-09-17T09:56:26.939 INFO:teuthology.orchestra.run.smithi018.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,214 kB] 2021-09-17T09:56:27.071 INFO:teuthology.orchestra.run.smithi018.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [259 kB] 2021-09-17T09:56:27.090 INFO:teuthology.orchestra.run.smithi018.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.3 kB] 2021-09-17T09:56:27.092 INFO:teuthology.orchestra.run.smithi018.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [443 kB] 2021-09-17T09:56:27.127 INFO:teuthology.orchestra.run.smithi018.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [19.3 kB] 2021-09-17T09:56:27.127 INFO:teuthology.orchestra.run.smithi018.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [63.5 kB] 2021-09-17T09:56:27.128 INFO:teuthology.orchestra.run.smithi018.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [504 B] 2021-09-17T09:56:27.129 INFO:teuthology.orchestra.run.smithi018.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [855 kB] 2021-09-17T09:56:27.156 INFO:teuthology.orchestra.run.smithi018.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [633 kB] 2021-09-17T09:56:27.187 INFO:teuthology.orchestra.run.smithi018.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [182 kB] 2021-09-17T09:56:27.196 INFO:teuthology.orchestra.run.smithi018.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [18.9 kB] 2021-09-17T09:56:27.212 INFO:teuthology.orchestra.run.smithi018.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,232 B] 2021-09-17T09:56:27.212 INFO:teuthology.orchestra.run.smithi018.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.6 kB] 2021-09-17T09:56:27.213 INFO:teuthology.orchestra.run.smithi018.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [6,776 B] 2021-09-17T09:56:27.214 INFO:teuthology.orchestra.run.smithi018.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2021-09-17T09:56:27.214 INFO:teuthology.orchestra.run.smithi018.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [5,800 B] 2021-09-17T09:56:27.238 INFO:teuthology.orchestra.run.smithi018.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [4,736 B] 2021-09-17T09:56:27.323 INFO:teuthology.orchestra.run.smithi018.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [276 B] 2021-09-17T09:56:30.675 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 7,714 kB in 2s (4,898 kB/s) 2021-09-17T09:56:31.871 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2021-09-17T09:56:31.895 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-09-17T09:56:31.964 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2021-09-17T09:56:32.132 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2021-09-17T09:56:32.133 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2021-09-17T09:56:32.155 INFO:teuthology.orchestra.run.smithi018.stderr:E: Unable to locate package linux-image-current-generic 2021-09-17T09:56:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-09-17T09:56:32.157 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-09-17T09:56:32.224 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2021-09-17T09:56:32.395 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2021-09-17T09:56:32.396 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2021-09-17T09:56:32.548 INFO:teuthology.orchestra.run.smithi018.stdout:The following additional packages will be installed: 2021-09-17T09:56:32.548 INFO:teuthology.orchestra.run.smithi018.stdout: linux-generic linux-headers-5.4.0-84 linux-headers-5.4.0-84-generic 2021-09-17T09:56:32.548 INFO:teuthology.orchestra.run.smithi018.stdout: linux-headers-generic linux-image-5.4.0-84-generic 2021-09-17T09:56:32.549 INFO:teuthology.orchestra.run.smithi018.stdout: linux-modules-5.4.0-84-generic linux-modules-extra-5.4.0-84-generic 2021-09-17T09:56:32.549 INFO:teuthology.orchestra.run.smithi018.stdout:Suggested packages: 2021-09-17T09:56:32.550 INFO:teuthology.orchestra.run.smithi018.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2021-09-17T09:56:32.550 INFO:teuthology.orchestra.run.smithi018.stdout:Recommended packages: 2021-09-17T09:56:32.550 INFO:teuthology.orchestra.run.smithi018.stdout: thermald 2021-09-17T09:56:32.578 INFO:teuthology.orchestra.run.smithi018.stdout:The following NEW packages will be installed: 2021-09-17T09:56:32.579 INFO:teuthology.orchestra.run.smithi018.stdout: linux-headers-5.4.0-84 linux-headers-5.4.0-84-generic 2021-09-17T09:56:32.579 INFO:teuthology.orchestra.run.smithi018.stdout: linux-image-5.4.0-84-generic linux-modules-5.4.0-84-generic 2021-09-17T09:56:32.579 INFO:teuthology.orchestra.run.smithi018.stdout: linux-modules-extra-5.4.0-84-generic 2021-09-17T09:56:32.580 INFO:teuthology.orchestra.run.smithi018.stdout:The following packages will be upgraded: 2021-09-17T09:56:32.580 INFO:teuthology.orchestra.run.smithi018.stdout: linux-generic linux-headers-generic linux-image-generic 2021-09-17T09:56:32.777 INFO:teuthology.orchestra.run.smithi018.stdout:3 upgraded, 5 newly installed, 0 to remove and 177 not upgraded. 2021-09-17T09:56:32.777 INFO:teuthology.orchestra.run.smithi018.stdout:Need to get 75.6 MB of archives. 2021-09-17T09:56:32.777 INFO:teuthology.orchestra.run.smithi018.stdout:After this operation, 378 MB of additional disk space will be used. 2021-09-17T09:56:32.777 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-84-generic amd64 5.4.0-84.94 [14.8 MB] 2021-09-17T09:56:33.995 INFO:teuthology.orchestra.run.smithi018.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-84-generic amd64 5.4.0-84.94 [8,992 kB] 2021-09-17T09:56:34.223 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-84-generic amd64 5.4.0-84.94 [39.4 MB] 2021-09-17T09:56:35.615 INFO:teuthology.orchestra.run.smithi018.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.84.88 [1,896 B] 2021-09-17T09:56:35.616 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.84.88 [2,568 B] 2021-09-17T09:56:35.616 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-84 all 5.4.0-84.94 [11.0 MB] 2021-09-17T09:56:36.081 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-84-generic amd64 5.4.0-84.94 [1,396 kB] 2021-09-17T09:56:36.283 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.84.88 [2,436 B] 2021-09-17T09:56:36.699 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 75.6 MB in 4s (20.4 MB/s) 2021-09-17T09:56:36.858 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-modules-5.4.0-84-generic. 2021-09-17T09:56:38.376 INFO:teuthology.orchestra.run.smithi018.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101688 files and directories currently installed.) 2021-09-17T09:56:38.381 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../0-linux-modules-5.4.0-84-generic_5.4.0-84.94_amd64.deb ... 2021-09-17T09:56:38.529 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-modules-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:56:40.870 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-image-5.4.0-84-generic. 2021-09-17T09:56:40.886 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../1-linux-image-5.4.0-84-generic_5.4.0-84.94_amd64.deb ... 2021-09-17T09:56:40.993 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-image-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:56:42.012 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-84-generic. 2021-09-17T09:56:42.028 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-84-generic_5.4.0-84.94_amd64.deb ... 2021-09-17T09:56:42.086 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-modules-extra-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:56:47.524 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../3-linux-generic_5.4.0.84.88_amd64.deb ... 2021-09-17T09:56:47.664 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-generic (5.4.0.84.88) over (5.4.0.66.69) ... 2021-09-17T09:56:48.042 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.84.88_amd64.deb ... 2021-09-17T09:56:48.181 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-image-generic (5.4.0.84.88) over (5.4.0.66.69) ... 2021-09-17T09:56:48.513 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-headers-5.4.0-84. 2021-09-17T09:56:48.529 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../5-linux-headers-5.4.0-84_5.4.0-84.94_all.deb ... 2021-09-17T09:56:48.563 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-headers-5.4.0-84 (5.4.0-84.94) ... 2021-09-17T09:56:55.202 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-headers-5.4.0-84-generic. 2021-09-17T09:56:55.224 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../6-linux-headers-5.4.0-84-generic_5.4.0-84.94_amd64.deb ... 2021-09-17T09:56:55.268 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-headers-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:56:58.085 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.84.88_amd64.deb ... 2021-09-17T09:56:58.228 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-headers-generic (5.4.0.84.88) over (5.4.0.66.69) ... 2021-09-17T09:56:58.605 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-headers-5.4.0-84 (5.4.0-84.94) ... 2021-09-17T09:56:58.724 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-modules-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:57:00.001 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-headers-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:57:00.152 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-headers-generic (5.4.0.84.88) ... 2021-09-17T09:57:00.306 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-image-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:57:01.559 INFO:teuthology.orchestra.run.smithi018.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-84-generic 2021-09-17T09:57:01.560 INFO:teuthology.orchestra.run.smithi018.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-84-generic 2021-09-17T09:57:01.706 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-modules-extra-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:57:02.949 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-image-generic (5.4.0.84.88) ... 2021-09-17T09:57:03.075 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-generic (5.4.0.84.88) ... 2021-09-17T09:57:03.226 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for linux-image-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:57:03.436 INFO:teuthology.orchestra.run.smithi018.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-09-17T09:57:03.436 INFO:teuthology.orchestra.run.smithi018.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-84-generic 2021-09-17T09:57:25.510 INFO:teuthology.orchestra.run.smithi018.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-09-17T09:57:25.511 INFO:teuthology.orchestra.run.smithi018.stdout:Sourcing file `/etc/default/grub' 2021-09-17T09:57:25.517 INFO:teuthology.orchestra.run.smithi018.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2021-09-17T09:57:25.539 INFO:teuthology.orchestra.run.smithi018.stdout:Generating grub configuration file ... 2021-09-17T09:57:25.853 INFO:teuthology.orchestra.run.smithi018.stdout:Found linux image: /boot/vmlinuz-5.4.0-84-generic 2021-09-17T09:57:25.874 INFO:teuthology.orchestra.run.smithi018.stdout:Found initrd image: /boot/initrd.img-5.4.0-84-generic 2021-09-17T09:57:26.275 INFO:teuthology.orchestra.run.smithi018.stdout:Found linux image: /boot/vmlinuz-5.4.0-66-generic 2021-09-17T09:57:26.287 INFO:teuthology.orchestra.run.smithi018.stdout:Found initrd image: /boot/initrd.img-5.4.0-66-generic 2021-09-17T09:57:28.861 INFO:teuthology.orchestra.run.smithi018.stdout:done 2021-09-17T09:57:29.265 DEBUG:teuthology.orchestra.run.smithi018:> dpkg -s linux-image-generic 2021-09-17T09:57:29.291 INFO:teuthology.orchestra.run.smithi018.stdout:Package: linux-image-generic 2021-09-17T09:57:29.292 INFO:teuthology.orchestra.run.smithi018.stdout:Status: install ok installed 2021-09-17T09:57:29.292 INFO:teuthology.orchestra.run.smithi018.stdout:Priority: optional 2021-09-17T09:57:29.293 INFO:teuthology.orchestra.run.smithi018.stdout:Section: kernel 2021-09-17T09:57:29.293 INFO:teuthology.orchestra.run.smithi018.stdout:Installed-Size: 18 2021-09-17T09:57:29.293 INFO:teuthology.orchestra.run.smithi018.stdout:Maintainer: Ubuntu Kernel Team 2021-09-17T09:57:29.294 INFO:teuthology.orchestra.run.smithi018.stdout:Architecture: amd64 2021-09-17T09:57:29.294 INFO:teuthology.orchestra.run.smithi018.stdout:Source: linux-meta 2021-09-17T09:57:29.294 INFO:teuthology.orchestra.run.smithi018.stdout:Version: 5.4.0.84.88 2021-09-17T09:57:29.295 INFO:teuthology.orchestra.run.smithi018.stdout:Provides: virtualbox-guest-modules (= 6.1.22-dfsg-2~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.12) 2021-09-17T09:57:29.295 INFO:teuthology.orchestra.run.smithi018.stdout:Depends: linux-image-5.4.0-84-generic, linux-modules-extra-5.4.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2021-09-17T09:57:29.296 INFO:teuthology.orchestra.run.smithi018.stdout:Recommends: thermald 2021-09-17T09:57:29.296 INFO:teuthology.orchestra.run.smithi018.stdout:Description: Generic Linux kernel image 2021-09-17T09:57:29.296 INFO:teuthology.orchestra.run.smithi018.stdout: This package will always depend on the latest generic kernel image 2021-09-17T09:57:29.297 INFO:teuthology.orchestra.run.smithi018.stdout: available. 2021-09-17T09:57:29.297 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-66-generic Expected: 5.4.0-84-generic 2021-09-17T09:57:29.298 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2021-09-17T09:57:29.305 INFO:teuthology.orchestra.run.smithi050.stdout:5.4.0-66-generic 2021-09-17T09:57:29.306 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.4.0-66-generic 2021-09-17T09:57:29.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get clean 2021-09-17T09:57:29.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get update 2021-09-17T09:57:29.741 INFO:teuthology.orchestra.run.smithi050.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2021-09-17T09:57:29.852 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-09-17T09:57:29.935 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2021-09-17T09:57:30.121 INFO:teuthology.orchestra.run.smithi050.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [867 kB] 2021-09-17T09:57:30.231 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [282 kB] 2021-09-17T09:57:30.246 INFO:teuthology.orchestra.run.smithi050.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [167 kB] 2021-09-17T09:57:30.246 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [8,712 B] 2021-09-17T09:57:30.247 INFO:teuthology.orchestra.run.smithi050.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [18.0 kB] 2021-09-17T09:57:30.247 INFO:teuthology.orchestra.run.smithi050.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [406 kB] 2021-09-17T09:57:30.248 INFO:teuthology.orchestra.run.smithi050.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [58.3 kB] 2021-09-17T09:57:30.251 INFO:teuthology.orchestra.run.smithi050.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [508 B] 2021-09-17T09:57:30.251 INFO:teuthology.orchestra.run.smithi050.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [507 kB] 2021-09-17T09:57:30.259 INFO:teuthology.orchestra.run.smithi050.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [639 kB] 2021-09-17T09:57:30.265 INFO:teuthology.orchestra.run.smithi050.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [102 kB] 2021-09-17T09:57:30.267 INFO:teuthology.orchestra.run.smithi050.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [12.4 kB] 2021-09-17T09:57:30.268 INFO:teuthology.orchestra.run.smithi050.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [21.9 kB] 2021-09-17T09:57:30.268 INFO:teuthology.orchestra.run.smithi050.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [4,948 B] 2021-09-17T09:57:30.274 INFO:teuthology.orchestra.run.smithi050.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB] 2021-09-17T09:57:30.287 INFO:teuthology.orchestra.run.smithi050.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2021-09-17T09:57:30.447 INFO:teuthology.orchestra.run.smithi050.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [536 kB] 2021-09-17T09:57:30.679 INFO:teuthology.orchestra.run.smithi050.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,214 kB] 2021-09-17T09:57:30.820 INFO:teuthology.orchestra.run.smithi050.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [259 kB] 2021-09-17T09:57:30.841 INFO:teuthology.orchestra.run.smithi050.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.3 kB] 2021-09-17T09:57:30.842 INFO:teuthology.orchestra.run.smithi050.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [19.3 kB] 2021-09-17T09:57:30.844 INFO:teuthology.orchestra.run.smithi050.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [443 kB] 2021-09-17T09:57:30.862 INFO:teuthology.orchestra.run.smithi050.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [63.5 kB] 2021-09-17T09:57:30.865 INFO:teuthology.orchestra.run.smithi050.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [504 B] 2021-09-17T09:57:30.865 INFO:teuthology.orchestra.run.smithi050.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [633 kB] 2021-09-17T09:57:30.895 INFO:teuthology.orchestra.run.smithi050.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [855 kB] 2021-09-17T09:57:30.929 INFO:teuthology.orchestra.run.smithi050.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [182 kB] 2021-09-17T09:57:30.935 INFO:teuthology.orchestra.run.smithi050.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [18.9 kB] 2021-09-17T09:57:30.947 INFO:teuthology.orchestra.run.smithi050.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,232 B] 2021-09-17T09:57:30.948 INFO:teuthology.orchestra.run.smithi050.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.6 kB] 2021-09-17T09:57:30.948 INFO:teuthology.orchestra.run.smithi050.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [6,776 B] 2021-09-17T09:57:30.949 INFO:teuthology.orchestra.run.smithi050.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2021-09-17T09:57:30.949 INFO:teuthology.orchestra.run.smithi050.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [5,800 B] 2021-09-17T09:57:30.985 INFO:teuthology.orchestra.run.smithi050.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [4,736 B] 2021-09-17T09:57:30.985 INFO:teuthology.orchestra.run.smithi050.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [276 B] 2021-09-17T09:57:34.238 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 7,714 kB in 2s (4,798 kB/s) 2021-09-17T09:57:35.354 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2021-09-17T09:57:35.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-09-17T09:57:35.444 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2021-09-17T09:57:35.598 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2021-09-17T09:57:35.599 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2021-09-17T09:57:35.618 INFO:teuthology.orchestra.run.smithi050.stderr:E: Unable to locate package linux-image-current-generic 2021-09-17T09:57:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-09-17T09:57:35.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-09-17T09:57:35.690 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2021-09-17T09:57:35.840 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2021-09-17T09:57:35.841 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2021-09-17T09:57:35.977 INFO:teuthology.orchestra.run.smithi050.stdout:The following additional packages will be installed: 2021-09-17T09:57:35.977 INFO:teuthology.orchestra.run.smithi050.stdout: linux-generic linux-headers-5.4.0-84 linux-headers-5.4.0-84-generic 2021-09-17T09:57:35.978 INFO:teuthology.orchestra.run.smithi050.stdout: linux-headers-generic linux-image-5.4.0-84-generic 2021-09-17T09:57:35.978 INFO:teuthology.orchestra.run.smithi050.stdout: linux-modules-5.4.0-84-generic linux-modules-extra-5.4.0-84-generic 2021-09-17T09:57:35.979 INFO:teuthology.orchestra.run.smithi050.stdout:Suggested packages: 2021-09-17T09:57:35.979 INFO:teuthology.orchestra.run.smithi050.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2021-09-17T09:57:35.979 INFO:teuthology.orchestra.run.smithi050.stdout:Recommended packages: 2021-09-17T09:57:35.979 INFO:teuthology.orchestra.run.smithi050.stdout: thermald 2021-09-17T09:57:36.007 INFO:teuthology.orchestra.run.smithi050.stdout:The following NEW packages will be installed: 2021-09-17T09:57:36.007 INFO:teuthology.orchestra.run.smithi050.stdout: linux-headers-5.4.0-84 linux-headers-5.4.0-84-generic 2021-09-17T09:57:36.008 INFO:teuthology.orchestra.run.smithi050.stdout: linux-image-5.4.0-84-generic linux-modules-5.4.0-84-generic 2021-09-17T09:57:36.008 INFO:teuthology.orchestra.run.smithi050.stdout: linux-modules-extra-5.4.0-84-generic 2021-09-17T09:57:36.009 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages will be upgraded: 2021-09-17T09:57:36.009 INFO:teuthology.orchestra.run.smithi050.stdout: linux-generic linux-headers-generic linux-image-generic 2021-09-17T09:57:36.214 INFO:teuthology.orchestra.run.smithi050.stdout:3 upgraded, 5 newly installed, 0 to remove and 177 not upgraded. 2021-09-17T09:57:36.214 INFO:teuthology.orchestra.run.smithi050.stdout:Need to get 75.6 MB of archives. 2021-09-17T09:57:36.215 INFO:teuthology.orchestra.run.smithi050.stdout:After this operation, 378 MB of additional disk space will be used. 2021-09-17T09:57:36.215 INFO:teuthology.orchestra.run.smithi050.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-84-generic amd64 5.4.0-84.94 [14.8 MB] 2021-09-17T09:57:37.895 INFO:teuthology.orchestra.run.smithi050.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-84-generic amd64 5.4.0-84.94 [8,992 kB] 2021-09-17T09:57:38.170 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-84-generic amd64 5.4.0-84.94 [39.4 MB] 2021-09-17T09:57:39.555 INFO:teuthology.orchestra.run.smithi050.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.84.88 [1,896 B] 2021-09-17T09:57:39.556 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.84.88 [2,568 B] 2021-09-17T09:57:39.556 INFO:teuthology.orchestra.run.smithi050.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-84 all 5.4.0-84.94 [11.0 MB] 2021-09-17T09:57:39.862 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-84-generic amd64 5.4.0-84.94 [1,396 kB] 2021-09-17T09:57:39.900 INFO:teuthology.orchestra.run.smithi050.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.84.88 [2,436 B] 2021-09-17T09:57:40.296 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 75.6 MB in 4s (19.4 MB/s) 2021-09-17T09:57:40.460 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-modules-5.4.0-84-generic. 2021-09-17T09:57:41.961 INFO:teuthology.orchestra.run.smithi050.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101688 files and directories currently installed.) 2021-09-17T09:57:41.965 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../0-linux-modules-5.4.0-84-generic_5.4.0-84.94_amd64.deb ... 2021-09-17T09:57:42.023 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-modules-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:57:43.942 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-image-5.4.0-84-generic. 2021-09-17T09:57:43.957 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../1-linux-image-5.4.0-84-generic_5.4.0-84.94_amd64.deb ... 2021-09-17T09:57:44.073 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-image-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:57:45.083 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-84-generic. 2021-09-17T09:57:45.099 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-84-generic_5.4.0-84.94_amd64.deb ... 2021-09-17T09:57:45.144 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-modules-extra-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:57:50.211 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../3-linux-generic_5.4.0.84.88_amd64.deb ... 2021-09-17T09:57:50.330 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-generic (5.4.0.84.88) over (5.4.0.66.69) ... 2021-09-17T09:57:50.657 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.84.88_amd64.deb ... 2021-09-17T09:57:50.777 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-image-generic (5.4.0.84.88) over (5.4.0.66.69) ... 2021-09-17T09:57:51.126 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-headers-5.4.0-84. 2021-09-17T09:57:51.140 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../5-linux-headers-5.4.0-84_5.4.0-84.94_all.deb ... 2021-09-17T09:57:51.185 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-headers-5.4.0-84 (5.4.0-84.94) ... 2021-09-17T09:57:57.234 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-headers-5.4.0-84-generic. 2021-09-17T09:57:57.257 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../6-linux-headers-5.4.0-84-generic_5.4.0-84.94_amd64.deb ... 2021-09-17T09:57:57.300 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-headers-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:57:59.803 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.84.88_amd64.deb ... 2021-09-17T09:57:59.957 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-headers-generic (5.4.0.84.88) over (5.4.0.66.69) ... 2021-09-17T09:58:01.623 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-headers-5.4.0-84 (5.4.0-84.94) ... 2021-09-17T09:58:01.737 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-modules-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:58:02.955 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-headers-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:58:03.081 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-headers-generic (5.4.0.84.88) ... 2021-09-17T09:58:03.207 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-image-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:58:04.383 INFO:teuthology.orchestra.run.smithi050.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-84-generic 2021-09-17T09:58:04.383 INFO:teuthology.orchestra.run.smithi050.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-84-generic 2021-09-17T09:58:04.510 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-modules-extra-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:58:05.677 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-image-generic (5.4.0.84.88) ... 2021-09-17T09:58:05.778 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-generic (5.4.0.84.88) ... 2021-09-17T09:58:05.905 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for linux-image-5.4.0-84-generic (5.4.0-84.94) ... 2021-09-17T09:58:06.083 INFO:teuthology.orchestra.run.smithi050.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-09-17T09:58:06.084 INFO:teuthology.orchestra.run.smithi050.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-84-generic 2021-09-17T09:58:28.099 INFO:teuthology.orchestra.run.smithi050.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-09-17T09:58:28.100 INFO:teuthology.orchestra.run.smithi050.stdout:Sourcing file `/etc/default/grub' 2021-09-17T09:58:28.108 INFO:teuthology.orchestra.run.smithi050.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2021-09-17T09:58:28.130 INFO:teuthology.orchestra.run.smithi050.stdout:Generating grub configuration file ... 2021-09-17T09:58:28.460 INFO:teuthology.orchestra.run.smithi050.stdout:Found linux image: /boot/vmlinuz-5.4.0-84-generic 2021-09-17T09:58:28.480 INFO:teuthology.orchestra.run.smithi050.stdout:Found initrd image: /boot/initrd.img-5.4.0-84-generic 2021-09-17T09:58:28.878 INFO:teuthology.orchestra.run.smithi050.stdout:Found linux image: /boot/vmlinuz-5.4.0-66-generic 2021-09-17T09:58:28.890 INFO:teuthology.orchestra.run.smithi050.stdout:Found initrd image: /boot/initrd.img-5.4.0-66-generic 2021-09-17T09:58:31.181 INFO:teuthology.orchestra.run.smithi050.stdout:done 2021-09-17T09:58:31.569 DEBUG:teuthology.orchestra.run.smithi050:> dpkg -s linux-image-generic 2021-09-17T09:58:31.595 INFO:teuthology.orchestra.run.smithi050.stdout:Package: linux-image-generic 2021-09-17T09:58:31.596 INFO:teuthology.orchestra.run.smithi050.stdout:Status: install ok installed 2021-09-17T09:58:31.596 INFO:teuthology.orchestra.run.smithi050.stdout:Priority: optional 2021-09-17T09:58:31.596 INFO:teuthology.orchestra.run.smithi050.stdout:Section: kernel 2021-09-17T09:58:31.597 INFO:teuthology.orchestra.run.smithi050.stdout:Installed-Size: 18 2021-09-17T09:58:31.597 INFO:teuthology.orchestra.run.smithi050.stdout:Maintainer: Ubuntu Kernel Team 2021-09-17T09:58:31.597 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: amd64 2021-09-17T09:58:31.597 INFO:teuthology.orchestra.run.smithi050.stdout:Source: linux-meta 2021-09-17T09:58:31.597 INFO:teuthology.orchestra.run.smithi050.stdout:Version: 5.4.0.84.88 2021-09-17T09:58:31.598 INFO:teuthology.orchestra.run.smithi050.stdout:Provides: virtualbox-guest-modules (= 6.1.22-dfsg-2~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.12) 2021-09-17T09:58:31.598 INFO:teuthology.orchestra.run.smithi050.stdout:Depends: linux-image-5.4.0-84-generic, linux-modules-extra-5.4.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2021-09-17T09:58:31.598 INFO:teuthology.orchestra.run.smithi050.stdout:Recommends: thermald 2021-09-17T09:58:31.598 INFO:teuthology.orchestra.run.smithi050.stdout:Description: Generic Linux kernel image 2021-09-17T09:58:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: This package will always depend on the latest generic kernel image 2021-09-17T09:58:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: available. 2021-09-17T09:58:31.599 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-66-generic Expected: 5.4.0-84-generic 2021-09-17T09:58:31.600 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-09-17T09:58:31.600 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-09-17T09:58:31.600 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-09-17T09:58:31.601 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2021-09-17T09:58:31.601 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi018.front.sepia.ceph.com, path=None, version=distro) 2021-09-17T09:58:31.601 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get clean 2021-09-17T09:58:31.731 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get update 2021-09-17T09:58:31.875 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-09-17T09:58:31.982 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-09-17T09:58:32.065 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-09-17T09:58:32.147 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-09-17T09:58:33.408 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2021-09-17T09:58:33.429 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-09-17T09:58:33.494 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2021-09-17T09:58:33.644 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2021-09-17T09:58:33.645 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2021-09-17T09:58:33.667 INFO:teuthology.orchestra.run.smithi018.stderr:E: Unable to locate package linux-image-current-generic 2021-09-17T09:58:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-09-17T09:58:33.670 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-09-17T09:58:33.735 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2021-09-17T09:58:33.903 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2021-09-17T09:58:33.904 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2021-09-17T09:58:34.068 INFO:teuthology.orchestra.run.smithi018.stdout:linux-image-generic is already the newest version (5.4.0.84.88). 2021-09-17T09:58:34.068 INFO:teuthology.orchestra.run.smithi018.stdout:0 upgraded, 0 newly installed, 0 to remove and 177 not upgraded. 2021-09-17T09:58:34.070 DEBUG:teuthology.orchestra.run.smithi018:> dpkg -s linux-image-generic 2021-09-17T09:58:34.092 INFO:teuthology.orchestra.run.smithi018.stdout:Package: linux-image-generic 2021-09-17T09:58:34.092 INFO:teuthology.orchestra.run.smithi018.stdout:Status: install ok installed 2021-09-17T09:58:34.093 INFO:teuthology.orchestra.run.smithi018.stdout:Priority: optional 2021-09-17T09:58:34.093 INFO:teuthology.orchestra.run.smithi018.stdout:Section: kernel 2021-09-17T09:58:34.093 INFO:teuthology.orchestra.run.smithi018.stdout:Installed-Size: 18 2021-09-17T09:58:34.093 INFO:teuthology.orchestra.run.smithi018.stdout:Maintainer: Ubuntu Kernel Team 2021-09-17T09:58:34.093 INFO:teuthology.orchestra.run.smithi018.stdout:Architecture: amd64 2021-09-17T09:58:34.094 INFO:teuthology.orchestra.run.smithi018.stdout:Source: linux-meta 2021-09-17T09:58:34.094 INFO:teuthology.orchestra.run.smithi018.stdout:Version: 5.4.0.84.88 2021-09-17T09:58:34.094 INFO:teuthology.orchestra.run.smithi018.stdout:Provides: virtualbox-guest-modules (= 6.1.22-dfsg-2~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.12) 2021-09-17T09:58:34.094 INFO:teuthology.orchestra.run.smithi018.stdout:Depends: linux-image-5.4.0-84-generic, linux-modules-extra-5.4.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2021-09-17T09:58:34.095 INFO:teuthology.orchestra.run.smithi018.stdout:Recommends: thermald 2021-09-17T09:58:34.095 INFO:teuthology.orchestra.run.smithi018.stdout:Description: Generic Linux kernel image 2021-09-17T09:58:34.095 INFO:teuthology.orchestra.run.smithi018.stdout: This package will always depend on the latest generic kernel image 2021-09-17T09:58:34.095 INFO:teuthology.orchestra.run.smithi018.stdout: available. 2021-09-17T09:58:34.096 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2021-09-17T09:58:34.139 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.6bFHnI2Wzb 2021-09-17T09:58:34.140 DEBUG:teuthology.orchestra.run.smithi018:> sudo cp /boot/grub/grub.cfg /tmp/tmp.6bFHnI2Wzb 2021-09-17T09:58:34.159 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.6bFHnI2Wzb 2021-09-17T09:58:34.195 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.6bFHnI2Wzb is 8KB 2021-09-17T09:58:34.208 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.6bFHnI2Wzb 2021-09-17T09:58:34.215 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-09-17T09:58:34.272 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T09:58:34.272 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-09-17T09:58:34.273 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-09-17T09:58:34.321 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-84-generic 2021-09-17T09:58:34.321 DEBUG:teuthology.orchestra.run.smithi018:> sudo update-grub 2021-09-17T09:58:35.444 INFO:teuthology.orchestra.run.smithi018.stderr:Sourcing file `/etc/default/grub' 2021-09-17T09:58:35.445 INFO:teuthology.orchestra.run.smithi018.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2021-09-17T09:58:35.469 INFO:teuthology.orchestra.run.smithi018.stderr:Generating grub configuration file ... 2021-09-17T09:58:35.731 INFO:teuthology.orchestra.run.smithi018.stderr:Found linux image: /boot/vmlinuz-5.4.0-84-generic 2021-09-17T09:58:35.742 INFO:teuthology.orchestra.run.smithi018.stderr:Found initrd image: /boot/initrd.img-5.4.0-84-generic 2021-09-17T09:58:36.102 INFO:teuthology.orchestra.run.smithi018.stderr:Found linux image: /boot/vmlinuz-5.4.0-66-generic 2021-09-17T09:58:36.113 INFO:teuthology.orchestra.run.smithi018.stderr:Found initrd image: /boot/initrd.img-5.4.0-66-generic 2021-09-17T09:58:37.826 INFO:teuthology.orchestra.run.smithi018.stderr:done 2021-09-17T09:58:37.828 DEBUG:teuthology.orchestra.run.smithi018:> sudo shutdown -r now 2021-09-17T09:58:37.831 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2021-09-17T09:58:37.831 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2021-09-17T09:58:37.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get clean 2021-09-17T09:58:37.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get update 2021-09-17T09:58:38.041 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-09-17T09:58:38.153 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-09-17T09:58:38.235 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-09-17T09:58:38.318 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-09-17T09:58:39.571 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2021-09-17T09:58:39.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-09-17T09:58:39.660 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2021-09-17T09:58:39.826 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2021-09-17T09:58:39.836 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2021-09-17T09:58:39.860 INFO:teuthology.orchestra.run.smithi050.stderr:E: Unable to locate package linux-image-current-generic 2021-09-17T09:58:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-09-17T09:58:39.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-09-17T09:58:39.935 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2021-09-17T09:58:40.104 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2021-09-17T09:58:40.104 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2021-09-17T09:58:40.268 INFO:teuthology.orchestra.run.smithi050.stdout:linux-image-generic is already the newest version (5.4.0.84.88). 2021-09-17T09:58:40.269 INFO:teuthology.orchestra.run.smithi050.stdout:0 upgraded, 0 newly installed, 0 to remove and 177 not upgraded. 2021-09-17T09:58:40.271 DEBUG:teuthology.orchestra.run.smithi050:> dpkg -s linux-image-generic 2021-09-17T09:58:40.290 INFO:teuthology.orchestra.run.smithi050.stdout:Package: linux-image-generic 2021-09-17T09:58:40.290 INFO:teuthology.orchestra.run.smithi050.stdout:Status: install ok installed 2021-09-17T09:58:40.291 INFO:teuthology.orchestra.run.smithi050.stdout:Priority: optional 2021-09-17T09:58:40.291 INFO:teuthology.orchestra.run.smithi050.stdout:Section: kernel 2021-09-17T09:58:40.291 INFO:teuthology.orchestra.run.smithi050.stdout:Installed-Size: 18 2021-09-17T09:58:40.291 INFO:teuthology.orchestra.run.smithi050.stdout:Maintainer: Ubuntu Kernel Team 2021-09-17T09:58:40.291 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: amd64 2021-09-17T09:58:40.292 INFO:teuthology.orchestra.run.smithi050.stdout:Source: linux-meta 2021-09-17T09:58:40.292 INFO:teuthology.orchestra.run.smithi050.stdout:Version: 5.4.0.84.88 2021-09-17T09:58:40.292 INFO:teuthology.orchestra.run.smithi050.stdout:Provides: virtualbox-guest-modules (= 6.1.22-dfsg-2~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.12) 2021-09-17T09:58:40.292 INFO:teuthology.orchestra.run.smithi050.stdout:Depends: linux-image-5.4.0-84-generic, linux-modules-extra-5.4.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2021-09-17T09:58:40.293 INFO:teuthology.orchestra.run.smithi050.stdout:Recommends: thermald 2021-09-17T09:58:40.293 INFO:teuthology.orchestra.run.smithi050.stdout:Description: Generic Linux kernel image 2021-09-17T09:58:40.293 INFO:teuthology.orchestra.run.smithi050.stdout: This package will always depend on the latest generic kernel image 2021-09-17T09:58:40.293 INFO:teuthology.orchestra.run.smithi050.stdout: available. 2021-09-17T09:58:40.294 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2021-09-17T09:58:40.298 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.PNdwK6TQSO 2021-09-17T09:58:40.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub/grub.cfg /tmp/tmp.PNdwK6TQSO 2021-09-17T09:58:40.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.PNdwK6TQSO 2021-09-17T09:58:40.396 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.PNdwK6TQSO is 8KB 2021-09-17T09:58:40.410 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.PNdwK6TQSO 2021-09-17T09:58:40.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-09-17T09:58:40.476 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T09:58:40.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-09-17T09:58:40.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-09-17T09:58:40.526 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-84-generic 2021-09-17T09:58:40.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo update-grub 2021-09-17T09:58:41.629 INFO:teuthology.orchestra.run.smithi050.stderr:Sourcing file `/etc/default/grub' 2021-09-17T09:58:41.631 INFO:teuthology.orchestra.run.smithi050.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2021-09-17T09:58:41.645 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2021-09-17T09:58:41.891 INFO:teuthology.orchestra.run.smithi050.stderr:Found linux image: /boot/vmlinuz-5.4.0-84-generic 2021-09-17T09:58:41.903 INFO:teuthology.orchestra.run.smithi050.stderr:Found initrd image: /boot/initrd.img-5.4.0-84-generic 2021-09-17T09:58:42.240 INFO:teuthology.orchestra.run.smithi050.stderr:Found linux image: /boot/vmlinuz-5.4.0-66-generic 2021-09-17T09:58:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:Found initrd image: /boot/initrd.img-5.4.0-66-generic 2021-09-17T09:58:43.569 INFO:teuthology.orchestra.run.smithi050.stderr:done 2021-09-17T09:58:43.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2021-09-17T09:59:13.642 INFO:teuthology.misc:Re-opening connections... 2021-09-17T09:59:13.643 INFO:teuthology.misc:trying to connect to ubuntu@smithi018.front.sepia.ceph.com 2021-09-17T09:59:13.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2021-09-17T09:59:13.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-17T09:59:32.043 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.18 2021-09-17T09:59:32.044 INFO:teuthology.misc:trying to connect to ubuntu@smithi050.front.sepia.ceph.com 2021-09-17T09:59:32.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2021-09-17T09:59:32.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-17T10:00:32.104 DEBUG:teuthology.orchestra.remote:timed out 2021-09-17T10:00:32.106 DEBUG:teuthology.misc:waited 78.46254515647888 2021-09-17T10:00:33.107 INFO:teuthology.misc:trying to connect to ubuntu@smithi018.front.sepia.ceph.com 2021-09-17T10:00:33.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2021-09-17T10:00:33.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-17T10:00:33.294 DEBUG:teuthology.orchestra.run.smithi018:> true 2021-09-17T10:00:33.800 DEBUG:teuthology.misc:waited 80.15684223175049 2021-09-17T10:00:34.802 INFO:teuthology.misc:trying to connect to ubuntu@smithi050.front.sepia.ceph.com 2021-09-17T10:00:34.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2021-09-17T10:00:34.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-17T10:00:35.045 DEBUG:teuthology.orchestra.run.smithi050:> true 2021-09-17T10:00:35.596 DEBUG:teuthology.misc:waited 81.95247864723206 2021-09-17T10:00:36.597 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2021-09-17T10:00:36.598 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "5.4.0-84-generic"... 2021-09-17T10:00:36.598 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2021-09-17T10:00:36.606 INFO:teuthology.orchestra.run.smithi018.stdout:5.4.0-84-generic 2021-09-17T10:00:36.606 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-84-generic vs 5.4.0-84-generic 2021-09-17T10:00:36.606 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-09-17T10:00:36.607 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-09-17T10:00:36.607 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2021-09-17T10:00:36.608 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "5.4.0-84-generic"... 2021-09-17T10:00:36.608 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2021-09-17T10:00:36.616 INFO:teuthology.orchestra.run.smithi050.stdout:5.4.0-84-generic 2021-09-17T10:00:36.617 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-84-generic vs 5.4.0-84-generic 2021-09-17T10:00:36.617 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-09-17T10:00:36.617 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-09-17T10:00:37.618 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-09-17T10:00:37.619 DEBUG:teuthology.orchestra.run.smithi018:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-09-17T10:00:37.804 INFO:teuthology.orchestra.run.smithi018.stdout:ttyS1 2021-09-17T10:00:37.836 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-09-17T10:00:37.836 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-09-17T10:00:37.901 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2021-09-17T10:00:37.926 INFO:teuthology.run_tasks:Running task internal.base... 2021-09-17T10:00:37.938 INFO:teuthology.task.internal:Creating test directory... 2021-09-17T10:00:37.938 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-09-17T10:00:37.941 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-09-17T10:00:37.949 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-09-17T10:00:37.973 INFO:teuthology.run_tasks:Running task internal.archive... 2021-09-17T10:00:37.998 INFO:teuthology.task.internal:Creating archive directory... 2021-09-17T10:00:37.998 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-09-17T10:00:38.001 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-09-17T10:00:38.021 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-09-17T10:00:38.034 INFO:teuthology.task.internal:Enabling coredump saving... 2021-09-17T10:00:38.035 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-09-17T10:00:38.038 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-09-17T10:00:38.054 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-17T10:00:38.065 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-17T10:00:38.073 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-17T10:00:38.083 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-17T10:00:38.086 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-09-17T10:00:38.097 INFO:teuthology.task.internal:Configuring sudo... 2021-09-17T10:00:38.098 DEBUG:teuthology.orchestra.run.smithi018:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-09-17T10:00:38.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-09-17T10:00:38.129 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-09-17T10:00:38.140 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-09-17T10:00:38.141 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-09-17T10:00:38.143 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-09-17T10:00:38.150 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-17T10:00:38.156 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-17T10:00:38.205 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:00:38.205 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-09-17T10:00:38.242 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-17T10:00:38.250 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-17T10:00:38.258 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:00:38.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-09-17T10:00:38.285 DEBUG:teuthology.orchestra.run.smithi018:> sudo service rsyslog restart 2021-09-17T10:00:38.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2021-09-17T10:00:38.374 INFO:teuthology.run_tasks:Running task internal.timer... 2021-09-17T10:00:38.385 INFO:teuthology.task.internal:Starting timer... 2021-09-17T10:00:38.385 INFO:teuthology.run_tasks:Running task pcp... 2021-09-17T10:00:38.435 INFO:teuthology.run_tasks:Running task selinux... 2021-09-17T10:00:38.468 DEBUG:teuthology.task.selinux:Excluding smithi018: OS 'ubuntu' does not support SELinux 2021-09-17T10:00:38.468 DEBUG:teuthology.task.selinux:Excluding smithi050: OS 'ubuntu' does not support SELinux 2021-09-17T10:00:38.469 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-09-17T10:00:38.469 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-09-17T10:00:38.469 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-09-17T10:00:38.470 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-09-17T10:00:38.501 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-09-17T10:00:38.501 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-09-17T10:00:38.520 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-09-17T10:00:38.522 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi018.front.sepia.ceph.com,smithi050.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-09-17T10:00:38.900 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-09-17T10:00:39.713 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-09-17T10:00:42.649 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-09-17T10:00:42.672 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:42 +0000 (0:00:00.039) 0:00:00.039 ****** 2021-09-17T10:00:42.684 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:42 +0000 (0:00:00.012) 0:00:00.051 ****** 2021-09-17T10:00:44.968 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-09-17T10:00:44.970 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 20.04 on host smithi018.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi018.front.sepia.ceph.com] 2021-09-17T10:00:44.982 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 20.04 on host smithi050.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi050.front.sepia.ceph.com] 2021-09-17T10:00:45.044 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:45 +0000 (0:00:02.360) 0:00:02.412 ****** 2021-09-17T10:00:45.066 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:45 +0000 (0:00:00.021) 0:00:02.433 ****** 2021-09-17T10:00:45.261 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-09-17T10:00:45.263 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-09-17T10:00:45.285 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:45 +0000 (0:00:00.219) 0:00:02.653 ****** 2021-09-17T10:00:45.306 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:45 +0000 (0:00:00.020) 0:00:02.673 ****** 2021-09-17T10:00:45.804 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-09-17T10:00:45.806 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} ok: [smithi050.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-09-17T10:00:45.830 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:45 +0000 (0:00:00.524) 0:00:03.198 ****** 2021-09-17T10:00:45.851 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:45 +0000 (0:00:00.020) 0:00:03.218 ****** 2021-09-17T10:00:45.899 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-09-17T10:00:45.901 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:00:45.928 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:45 +0000 (0:00:00.077) 0:00:03.295 ****** 2021-09-17T10:00:45.935 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:00:45.961 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:45 +0000 (0:00:00.032) 0:00:03.328 ****** 2021-09-17T10:00:46.136 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-09-17T10:00:46.138 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi018.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T10:00:46.162 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:46 +0000 (0:00:00.201) 0:00:03.529 ****** 2021-09-17T10:00:46.190 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:46 +0000 (0:00:00.027) 0:00:03.557 ****** 2021-09-17T10:00:46.884 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "size": 182, "state": "file", "uid": 0} 2021-09-17T10:00:46.908 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:46 +0000 (0:00:00.717) 0:00:04.275 ****** 2021-09-17T10:00:46.916 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "size": 182, "state": "file", "uid": 0} 2021-09-17T10:00:46.917 INFO:teuthology.task.ansible.out: 2021-09-17T10:00:46.943 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:46 +0000 (0:00:00.035) 0:00:04.310 ****** 2021-09-17T10:00:47.272 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-09-17T10:00:47.274 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-09-17T10:00:47.276 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-09-17T10:00:47.318 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-09-17T10:00:47.320 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-09-17T10:00:47.387 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-09-17T10:00:47.389 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-17T10:00:47.403 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-09-17T10:00:47.405 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:47 +0000 (0:00:00.460) 0:00:04.771 ****** 2021-09-17T10:00:48.104 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] 2021-09-17T10:00:48.137 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] 2021-09-17T10:00:48.157 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** Friday 17 September 2021 10:00:48 +0000 (0:00:00.753) 0:00:05.525 ****** 2021-09-17T10:00:48.189 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-09-17T10:00:48.210 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-09-17T10:00:48.215 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-17T10:00:48.371 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:48 +0000 (0:00:00.214) 0:00:05.739 ****** 2021-09-17T10:00:48.394 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:48 +0000 (0:00:00.022) 0:00:05.761 ****** 2021-09-17T10:00:49.259 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-09-17T10:00:49.260 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] 2021-09-17T10:00:49.271 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] 2021-09-17T10:00:49.337 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-09-17T10:00:49.339 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:49 +0000 (0:00:00.943) 0:00:06.705 ****** 2021-09-17T10:00:49.362 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:49 +0000 (0:00:00.024) 0:00:06.730 ****** 2021-09-17T10:00:49.994 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-09-17T10:00:49.995 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.219239", "end": "2021-09-17 10:00:49.966928", "rc": 0, "start": "2021-09-17 10:00:49.747689", "stderr": "", "stderr_lines": [], "stdout": "39f530053dd0894c844c464d403644cc64956570", "stdout_lines": ["39f530053dd0894c844c464d403644cc64956570"]} 2021-09-17T10:00:50.021 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.658) 0:00:07.388 ****** 2021-09-17T10:00:50.028 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.226702", "end": "2021-09-17 10:00:49.996332", "rc": 0, "start": "2021-09-17 10:00:49.769630", "stderr": "", "stderr_lines": [], "stdout": "39f530053dd0894c844c464d403644cc64956570", "stdout_lines": ["39f530053dd0894c844c464d403644cc64956570"]} 2021-09-17T10:00:50.056 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.034) 0:00:07.423 ****** 2021-09-17T10:00:50.167 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-09-17T10:00:50.169 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013155", "end": "2021-09-17 10:00:51.446163", "failed_when_result": false, "rc": 0, "start": "2021-09-17 10:00:51.433008", "stderr": "", "stderr_lines": [], "stdout": "c29274177b79f72740fc714fa28b55400cd0737f", "stdout_lines": ["c29274177b79f72740fc714fa28b55400cd0737f"]} 2021-09-17T10:00:50.194 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.138) 0:00:07.561 ****** 2021-09-17T10:00:50.206 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010742", "end": "2021-09-17 10:00:49.816934", "failed_when_result": false, "rc": 0, "start": "2021-09-17 10:00:49.806192", "stderr": "", "stderr_lines": [], "stdout": "c29274177b79f72740fc714fa28b55400cd0737f", "stdout_lines": ["c29274177b79f72740fc714fa28b55400cd0737f"]} 2021-09-17T10:00:50.234 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.039) 0:00:07.601 ****** 2021-09-17T10:00:50.262 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-09-17T10:00:50.264 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:00:50.288 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.053) 0:00:07.655 ****** 2021-09-17T10:00:50.305 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:00:50.330 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.041) 0:00:07.697 ****** 2021-09-17T10:00:50.358 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-09-17T10:00:50.360 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:00:50.384 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.053) 0:00:07.751 ****** 2021-09-17T10:00:50.401 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:00:50.427 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.043) 0:00:07.795 ****** 2021-09-17T10:00:50.466 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-09-17T10:00:50.467 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:00:50.490 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.063) 0:00:07.858 ****** 2021-09-17T10:00:50.508 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:00:50.533 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.042) 0:00:07.901 ****** 2021-09-17T10:00:50.561 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-09-17T10:00:50.563 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:00:50.587 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.053) 0:00:07.954 ****** 2021-09-17T10:00:50.615 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:00:50.640 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.053) 0:00:08.007 ****** 2021-09-17T10:00:50.658 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-09-17T10:00:50.659 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:00:50.686 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.045) 0:00:08.053 ****** 2021-09-17T10:00:50.714 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:00:50.739 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.052) 0:00:08.106 ****** 2021-09-17T10:00:50.859 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-09-17T10:00:50.861 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T10:00:50.884 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.145) 0:00:08.252 ****** 2021-09-17T10:00:50.921 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T10:00:50.947 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:00:50 +0000 (0:00:00.062) 0:00:08.314 ****** 2021-09-17T10:00:51.120 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-09-17T10:00:51.122 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-09-17T10:00:51.184 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-09-17T10:00:51.271 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-09-17T10:00:51.335 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-09-17T10:00:51.419 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-09-17T10:00:51.472 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-09-17T10:00:51.567 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-09-17T10:00:51.619 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-09-17T10:00:51.714 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-09-17T10:00:51.767 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-09-17T10:00:51.861 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-09-17T10:00:51.914 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-09-17T10:00:52.008 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-09-17T10:00:52.052 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-09-17T10:00:52.146 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-09-17T10:00:52.200 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-09-17T10:00:52.295 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-09-17T10:00:52.338 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-09-17T10:00:52.443 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-09-17T10:00:52.497 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-09-17T10:00:52.592 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-09-17T10:00:52.644 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-09-17T10:00:52.739 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-09-17T10:00:52.781 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-09-17T10:00:52.887 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-09-17T10:00:52.931 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-09-17T10:00:53.026 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-09-17T10:00:53.069 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-09-17T10:00:53.173 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-09-17T10:00:53.216 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-09-17T10:00:53.320 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-09-17T10:00:53.363 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-09-17T10:00:53.468 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-09-17T10:00:53.501 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-09-17T10:00:53.617 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-09-17T10:00:53.648 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-09-17T10:00:53.753 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-09-17T10:00:53.788 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-09-17T10:00:53.902 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-09-17T10:00:53.934 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-09-17T10:00:54.049 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-09-17T10:00:54.071 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-09-17T10:00:54.187 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-09-17T10:00:54.219 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-09-17T10:00:54.334 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-09-17T10:00:54.367 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-09-17T10:00:54.482 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-09-17T10:00:54.516 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-09-17T10:00:54.622 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-09-17T10:00:54.665 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-09-17T10:00:54.781 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-09-17T10:00:54.814 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-09-17T10:00:54.919 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-09-17T10:00:54.952 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-09-17T10:00:55.067 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-09-17T10:00:55.099 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-09-17T10:00:55.214 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-09-17T10:00:55.256 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-09-17T10:00:55.361 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-09-17T10:00:55.404 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-09-17T10:00:55.508 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-09-17T10:00:55.541 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-09-17T10:00:55.656 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-09-17T10:00:55.689 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-09-17T10:00:55.793 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-09-17T10:00:55.836 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-09-17T10:00:55.941 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-17T10:00:55.973 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-17T10:00:56.088 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-17T10:00:56.120 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-17T10:00:56.224 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-09-17T10:00:56.270 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-09-17T10:00:56.375 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-09-17T10:00:56.418 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-09-17T10:00:56.512 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-09-17T10:00:56.555 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-09-17T10:00:56.661 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-09-17T10:00:56.703 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-09-17T10:00:56.809 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-09-17T10:00:56.852 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-09-17T10:00:56.957 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-09-17T10:00:57.000 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-09-17T10:00:57.115 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-17T10:00:57.137 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-17T10:00:57.262 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-17T10:00:57.294 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-17T10:00:57.400 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-09-17T10:00:57.442 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-09-17T10:00:57.548 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-09-17T10:00:57.580 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-09-17T10:00:57.696 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-09-17T10:00:57.728 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-09-17T10:00:57.845 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-09-17T10:00:57.867 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-09-17T10:00:57.988 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-09-17T10:00:58.020 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-09-17T10:00:58.136 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-09-17T10:00:58.169 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-09-17T10:00:58.274 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-09-17T10:00:58.316 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-09-17T10:00:58.421 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-09-17T10:00:58.464 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-09-17T10:00:58.569 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-09-17T10:00:58.611 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-09-17T10:00:58.717 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-09-17T10:00:58.759 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-09-17T10:00:58.854 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-09-17T10:00:58.908 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-09-17T10:00:59.004 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-09-17T10:00:59.046 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-09-17T10:00:59.151 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-09-17T10:00:59.194 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-09-17T10:00:59.301 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-09-17T10:00:59.344 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-09-17T10:00:59.449 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-09-17T10:00:59.492 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-09-17T10:00:59.598 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-09-17T10:00:59.641 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-09-17T10:00:59.736 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-17T10:00:59.788 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-17T10:00:59.883 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-09-17T10:00:59.925 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-09-17T10:01:00.030 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-09-17T10:01:00.072 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-09-17T10:01:00.177 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-09-17T10:01:00.220 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-09-17T10:01:00.315 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-09-17T10:01:00.368 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-09-17T10:01:00.462 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-09-17T10:01:00.515 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-09-17T10:01:00.609 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-09-17T10:01:00.662 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-09-17T10:01:00.748 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-09-17T10:01:00.801 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-09-17T10:01:00.895 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-09-17T10:01:00.949 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-09-17T10:01:01.043 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-09-17T10:01:01.086 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-09-17T10:01:01.191 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-09-17T10:01:01.234 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-09-17T10:01:01.328 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-09-17T10:01:01.382 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-09-17T10:01:01.476 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-09-17T10:01:01.520 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-09-17T10:01:01.625 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-09-17T10:01:01.679 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-09-17T10:01:01.764 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-09-17T10:01:01.817 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-09-17T10:01:01.912 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-09-17T10:01:01.967 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-09-17T10:01:02.052 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-09-17T10:01:02.105 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-09-17T10:01:02.201 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-09-17T10:01:02.254 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-09-17T10:01:02.349 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-09-17T10:01:02.401 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-09-17T10:01:02.496 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-09-17T10:01:02.550 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-09-17T10:01:02.634 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-09-17T10:01:02.698 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-09-17T10:01:02.783 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-09-17T10:01:02.837 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-09-17T10:01:02.931 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-09-17T10:01:02.984 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-09-17T10:01:03.078 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-09-17T10:01:03.134 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-09-17T10:01:03.218 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-09-17T10:01:03.292 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-09-17T10:01:03.366 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-09-17T10:01:03.429 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-09-17T10:01:03.504 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-09-17T10:01:03.590 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-09-17T10:01:03.653 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-09-17T10:01:03.748 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-09-17T10:01:03.801 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-09-17T10:01:03.886 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-09-17T10:01:03.964 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-09-17T10:01:04.037 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-09-17T10:01:04.111 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-09-17T10:01:04.175 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-09-17T10:01:04.270 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-09-17T10:01:04.323 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-09-17T10:01:04.408 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-09-17T10:01:04.471 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-09-17T10:01:04.555 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-09-17T10:01:04.618 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-09-17T10:01:04.702 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-09-17T10:01:04.765 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-09-17T10:01:04.849 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-09-17T10:01:04.912 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-09-17T10:01:04.986 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-17T10:01:05.051 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-17T10:01:05.136 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-09-17T10:01:05.200 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-09-17T10:01:05.284 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-09-17T10:01:05.348 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-09-17T10:01:05.435 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-09-17T10:01:05.499 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-09-17T10:01:05.583 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-09-17T10:01:05.646 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-09-17T10:01:05.730 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-09-17T10:01:05.783 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-09-17T10:01:05.869 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-09-17T10:01:05.935 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-09-17T10:01:06.026 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-09-17T10:01:06.090 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-09-17T10:01:06.176 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-09-17T10:01:06.239 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-09-17T10:01:06.242 INFO:teuthology.task.ansible.out: 2021-09-17T10:01:06.326 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-09-17T10:01:06.380 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-09-17T10:01:06.476 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-09-17T10:01:06.529 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-09-17T10:01:06.614 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-09-17T10:01:06.669 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-09-17T10:01:06.764 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-09-17T10:01:06.828 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-09-17T10:01:06.912 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-09-17T10:01:06.966 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-09-17T10:01:07.062 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-09-17T10:01:07.116 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-09-17T10:01:07.201 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-09-17T10:01:07.265 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-09-17T10:01:07.352 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-09-17T10:01:07.405 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-09-17T10:01:07.499 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-09-17T10:01:07.553 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-09-17T10:01:07.638 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-09-17T10:01:07.640 INFO:teuthology.task.ansible.out: 2021-09-17T10:01:07.701 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-09-17T10:01:07.786 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-09-17T10:01:07.849 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-09-17T10:01:07.933 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-09-17T10:01:07.999 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-09-17T10:01:08.084 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-09-17T10:01:08.138 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-09-17T10:01:08.224 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-09-17T10:01:08.288 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-09-17T10:01:08.374 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-09-17T10:01:08.438 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-09-17T10:01:08.512 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-09-17T10:01:08.596 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-09-17T10:01:08.659 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-09-17T10:01:08.732 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-09-17T10:01:08.806 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-09-17T10:01:08.870 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-09-17T10:01:08.956 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-09-17T10:01:09.020 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-09-17T10:01:09.105 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-09-17T10:01:09.168 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-09-17T10:01:09.242 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-09-17T10:01:09.316 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-09-17T10:01:09.390 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-09-17T10:01:09.453 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-09-17T10:01:09.538 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-09-17T10:01:09.601 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-09-17T10:01:09.685 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-09-17T10:01:09.748 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-09-17T10:01:09.832 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-09-17T10:01:09.886 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-09-17T10:01:09.980 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-09-17T10:01:10.038 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-09-17T10:01:10.133 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-09-17T10:01:10.175 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-09-17T10:01:10.281 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-09-17T10:01:10.324 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-09-17T10:01:10.429 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10103, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 10102} 2021-09-17T10:01:10.472 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10103, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 10102} 2021-09-17T10:01:10.576 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10104, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "move_home": false, "name": "amctagga", "shell": "/bin/bash", "state": "present", "uid": 10103} 2021-09-17T10:01:10.618 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10104, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "move_home": false, "name": "amctagga", "shell": "/bin/bash", "state": "present", "uid": 10103} 2021-09-17T10:01:10.734 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10105, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "move_home": false, "name": "kevinz", "shell": "/bin/bash", "state": "present", "uid": 10104} 2021-09-17T10:01:10.766 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10105, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "move_home": false, "name": "kevinz", "shell": "/bin/bash", "state": "present", "uid": 10104} 2021-09-17T10:01:10.882 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10106, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "move_home": false, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "uid": 10105} 2021-09-17T10:01:10.914 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10106, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "move_home": false, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "uid": 10105} 2021-09-17T10:01:11.020 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10107, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "move_home": false, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "uid": 10106} 2021-09-17T10:01:11.063 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10107, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "move_home": false, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "uid": 10106} 2021-09-17T10:01:11.160 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10108, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "move_home": false, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "uid": 10107} 2021-09-17T10:01:11.205 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10108, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "move_home": false, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "uid": 10107} 2021-09-17T10:01:11.322 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10109, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "move_home": false, "name": "lflores", "shell": "/bin/bash", "state": "present", "uid": 10108} 2021-09-17T10:01:11.355 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10109, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "move_home": false, "name": "lflores", "shell": "/bin/bash", "state": "present", "uid": 10108} 2021-09-17T10:01:11.460 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10110, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "move_home": false, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "uid": 10109} 2021-09-17T10:01:11.493 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10110, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "move_home": false, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "uid": 10109} 2021-09-17T10:01:11.609 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10111, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "move_home": false, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "uid": 10110} 2021-09-17T10:01:11.642 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10111, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "move_home": false, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "uid": 10110} 2021-09-17T10:01:11.747 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10112, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "move_home": false, "name": "luisrico", "shell": "/bin/bash", "state": "present", "uid": 10111} 2021-09-17T10:01:11.789 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10112, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "move_home": false, "name": "luisrico", "shell": "/bin/bash", "state": "present", "uid": 10111} 2021-09-17T10:01:11.895 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10113, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "move_home": false, "name": "matan", "shell": "/bin/bash", "state": "present", "uid": 10112} 2021-09-17T10:01:11.928 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10113, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "move_home": false, "name": "matan", "shell": "/bin/bash", "state": "present", "uid": 10112} 2021-09-17T10:01:12.054 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10114, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "move_home": false, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "uid": 10113} 2021-09-17T10:01:12.075 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10114, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "move_home": false, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "uid": 10113} 2021-09-17T10:01:12.191 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10115, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "move_home": false, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "uid": 10114} 2021-09-17T10:01:12.224 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10115, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "move_home": false, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "uid": 10114} 2021-09-17T10:01:12.338 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10116, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "move_home": false, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "uid": 10115} 2021-09-17T10:01:12.371 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10116, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "move_home": false, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "uid": 10115} 2021-09-17T10:01:12.486 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10117, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "move_home": false, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "uid": 10116} 2021-09-17T10:01:12.519 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10117, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "move_home": false, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "uid": 10116} 2021-09-17T10:01:12.624 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10118, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "move_home": false, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "uid": 10117} 2021-09-17T10:01:12.666 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10118, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "move_home": false, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "uid": 10117} 2021-09-17T10:01:12.772 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10119, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "move_home": false, "name": "melissali", "shell": "/bin/bash", "state": "present", "uid": 10118} 2021-09-17T10:01:12.815 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10119, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "move_home": false, "name": "melissali", "shell": "/bin/bash", "state": "present", "uid": 10118} 2021-09-17T10:01:13.388 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-09-17T10:01:13.420 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-09-17T10:01:13.827 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-09-17T10:01:13.839 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-09-17T10:01:14.286 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-09-17T10:01:14.381 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-09-17T10:01:14.819 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10123, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10122} 2021-09-17T10:01:15.057 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10123, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10122} 2021-09-17T10:01:15.160 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:15 +0000 (0:00:24.213) 0:00:32.527 ****** 2021-09-17T10:01:15.184 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:15 +0000 (0:00:00.023) 0:00:32.551 ****** 2021-09-17T10:01:15.227 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-09-17T10:01:15.252 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:15 +0000 (0:00:00.068) 0:00:32.619 ****** 2021-09-17T10:01:15.287 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:15 +0000 (0:00:00.034) 0:00:32.654 ****** 2021-09-17T10:01:15.339 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-09-17T10:01:15.341 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-09-17T10:01:15.343 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-09-17T10:01:15.345 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed208 2021-09-17T10:01:15.346 INFO:teuthology.task.ansible.out:2caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": 2021-09-17T10:01:15.348 INFO:teuthology.task.ansible.out: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d 2021-09-17T10:01:15.349 INFO:teuthology.task.ansible.out:3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDe 2021-09-17T10:01:15.351 INFO:teuthology.task.ansible.out:vAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A 2021-09-17T10:01:15.352 INFO:teuthology.task.ansible.out:a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBM 2021-09-17T10:01:15.352 INFO:teuthology.task.ansible.out:gw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmeri 2021-09-17T10:01:15.354 INFO:teuthology.task.ansible.out:cp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361 2021-09-17T10:01:15.356 INFO:teuthology.task.ansible.out:f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9 2021-09-17T10:01:15.357 INFO:teuthology.task.ansible.out:JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67" 2021-09-17T10:01:15.358 INFO:teuthology.task.ansible.out:}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": " 2021-09-17T10:01:15.360 INFO:teuthology.task.ansible.out:shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6 2021-09-17T10:01:15.362 INFO:teuthology.task.ansible.out:878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovp 2021-09-17T10:01:15.363 INFO:teuthology.task.ansible.out:n": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}]}, "changed": false} 2021-09-17T10:01:15.390 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:15 +0000 (0:00:00.102) 0:00:32.757 ****** 2021-09-17T10:01:15.403 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-09-17T10:01:15.405 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-09-17T10:01:15.407 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed208 2021-09-17T10:01:15.408 INFO:teuthology.task.ansible.out:2caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": 2021-09-17T10:01:15.410 INFO:teuthology.task.ansible.out: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebb 2021-09-17T10:01:15.410 INFO:teuthology.task.ansible.out:a6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d 2021-09-17T10:01:15.412 INFO:teuthology.task.ansible.out:3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDe 2021-09-17T10:01:15.413 INFO:teuthology.task.ansible.out:vAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBM 2021-09-17T10:01:15.415 INFO:teuthology.task.ansible.out:gw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmeri 2021-09-17T10:01:15.417 INFO:teuthology.task.ansible.out:cp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361 2021-09-17T10:01:15.418 INFO:teuthology.task.ansible.out:f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9 2021-09-17T10:01:15.420 INFO:teuthology.task.ansible.out:JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": " 2021-09-17T10:01:15.421 INFO:teuthology.task.ansible.out:shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6 2021-09-17T10:01:15.423 INFO:teuthology.task.ansible.out:878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631 2021-09-17T10:01:15.423 INFO:teuthology.task.ansible.out:f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovp 2021-09-17T10:01:15.425 INFO:teuthology.task.ansible.out:n": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}]}, "changed": false} 2021-09-17T10:01:15.454 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:15 +0000 (0:00:00.064) 0:00:32.822 ****** 2021-09-17T10:01:16.148 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-09-17T10:01:16.150 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"after": "39f530053dd0894c844c464d403644cc64956570", "attempts": 1, "before": "39f530053dd0894c844c464d403644cc64956570", "changed": false, "remote_url_changed": false} 2021-09-17T10:01:16.174 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:16 +0000 (0:00:00.719) 0:00:33.541 ****** 2021-09-17T10:01:16.181 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"after": "39f530053dd0894c844c464d403644cc64956570", "attempts": 1, "before": "39f530053dd0894c844c464d403644cc64956570", "changed": false, "remote_url_changed": false} 2021-09-17T10:01:16.207 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:16 +0000 (0:00:00.033) 0:00:33.574 ****** 2021-09-17T10:01:16.423 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-09-17T10:01:16.425 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-09-17T10:01:16.455 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-09-17T10:01:16.560 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-09-17T10:01:16.603 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-09-17T10:01:16.698 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-09-17T10:01:16.730 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-09-17T10:01:16.825 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-09-17T10:01:16.827 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-09-17T10:01:16.868 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-09-17T10:01:16.870 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-09-17T10:01:16.951 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-09-17T10:01:17.004 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-09-17T10:01:17.097 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-09-17T10:01:17.099 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-09-17T10:01:17.130 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-09-17T10:01:17.131 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-09-17T10:01:17.236 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-09-17T10:01:17.269 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-09-17T10:01:17.363 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-09-17T10:01:17.395 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-09-17T10:01:17.500 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-09-17T10:01:17.513 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:17.533 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-09-17T10:01:17.535 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:17.644 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-09-17T10:01:17.646 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-09-17T10:01:17.665 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-09-17T10:01:17.667 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-09-17T10:01:17.771 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-09-17T10:01:17.773 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-09-17T10:01:17.782 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:17.793 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:17.804 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-09-17T10:01:17.807 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-09-17T10:01:17.815 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:17.816 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:17.934 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-09-17T10:01:17.955 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-09-17T10:01:18.075 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-09-17T10:01:18.078 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-09-17T10:01:18.087 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-09-17T10:01:18.089 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-09-17T10:01:18.205 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-09-17T10:01:18.216 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-09-17T10:01:18.353 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-09-17T10:01:18.364 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-09-17T10:01:18.490 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-09-17T10:01:18.501 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-09-17T10:01:18.628 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-09-17T10:01:18.629 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-09-17T10:01:18.639 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-09-17T10:01:18.641 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-09-17T10:01:18.766 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-09-17T10:01:18.768 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJo 2021-09-17T10:01:18.770 INFO:teuthology.task.ansible.out:SrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-09-17T10:01:18.905 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-09-17T10:01:18.917 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-09-17T10:01:19.045 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-09-17T10:01:19.046 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-09-17T10:01:19.194 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlI 2021-09-17T10:01:19.196 INFO:teuthology.task.ansible.out:Qv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-09-17T10:01:19.217 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlI 2021-09-17T10:01:19.219 INFO:teuthology.task.ansible.out:Qv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-09-17T10:01:19.343 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-09-17T10:01:19.481 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-09-17T10:01:19.484 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-09-17T10:01:19.607 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-09-17T10:01:19.618 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-09-17T10:01:19.755 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-09-17T10:01:19.756 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-09-17T10:01:19.883 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-09-17T10:01:19.894 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-09-17T10:01:20.021 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-09-17T10:01:20.023 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-09-17T10:01:20.032 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-09-17T10:01:20.033 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-09-17T10:01:20.155 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-09-17T10:01:20.177 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-09-17T10:01:20.305 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-09-17T10:01:20.307 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-09-17T10:01:20.443 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-09-17T10:01:20.445 INFO:teuthology.task.ansible.out: ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-09-17T10:01:20.570 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-09-17T10:01:20.581 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-09-17T10:01:20.707 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-17T10:01:20.718 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-17T10:01:20.845 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-09-17T10:01:20.847 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-09-17T10:01:20.981 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-09-17T10:01:20.983 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-09-17T10:01:21.108 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-09-17T10:01:21.110 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6 2021-09-17T10:01:21.112 INFO:teuthology.task.ansible.out:UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-09-17T10:01:21.245 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-09-17T10:01:21.247 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-09-17T10:01:21.256 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-09-17T10:01:21.258 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-09-17T10:01:21.382 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-09-17T10:01:21.386 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-09-17T10:01:21.509 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-09-17T10:01:21.520 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-09-17T10:01:21.647 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-09-17T10:01:21.649 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-09-17T10:01:21.789 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-09-17T10:01:21.791 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-17T10:01:21.800 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-09-17T10:01:21.802 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-17T10:01:21.927 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-09-17T10:01:21.929 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-09-17T10:01:21.930 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMS 2021-09-17T10:01:21.932 INFO:teuthology.task.ansible.out:woj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf 2021-09-17T10:01:21.934 INFO:teuthology.task.ansible.out:53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-09-17T10:01:22.064 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-09-17T10:01:22.066 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-09-17T10:01:22.192 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-09-17T10:01:22.204 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-09-17T10:01:22.330 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-09-17T10:01:22.331 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-09-17T10:01:22.466 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-09-17T10:01:22.467 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-09-17T10:01:22.592 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-09-17T10:01:22.603 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-09-17T10:01:22.732 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-09-17T10:01:22.765 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0b 2021-09-17T10:01:22.767 INFO:teuthology.task.ansible.out:sfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-09-17T10:01:22.873 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-09-17T10:01:22.898 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-09-17T10:01:23.000 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-09-17T10:01:23.039 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-09-17T10:01:23.144 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-09-17T10:01:23.147 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-09-17T10:01:23.168 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-09-17T10:01:23.170 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-09-17T10:01:23.284 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-09-17T10:01:23.306 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-09-17T10:01:23.421 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-09-17T10:01:23.443 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-09-17T10:01:23.549 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-09-17T10:01:23.551 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-09-17T10:01:23.582 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-09-17T10:01:23.584 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-09-17T10:01:23.687 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-09-17T10:01:23.720 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-09-17T10:01:23.815 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-09-17T10:01:23.847 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-09-17T10:01:23.952 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-09-17T10:01:23.996 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-09-17T10:01:24.080 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-17T10:01:24.123 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-17T10:01:24.218 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-09-17T10:01:24.281 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-09-17T10:01:24.344 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-09-17T10:01:24.419 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-09-17T10:01:24.482 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-09-17T10:01:24.556 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-09-17T10:01:24.620 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-09-17T10:01:24.684 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-09-17T10:01:24.747 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-09-17T10:01:24.822 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-09-17T10:01:24.889 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-09-17T10:01:24.953 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-09-17T10:01:25.017 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-09-17T10:01:25.091 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-09-17T10:01:25.146 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-09-17T10:01:25.219 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-09-17T10:01:25.283 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-09-17T10:01:25.346 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-09-17T10:01:25.422 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-09-17T10:01:25.496 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-09-17T10:01:25.569 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-09-17T10:01:25.644 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-09-17T10:01:25.696 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-09-17T10:01:25.791 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-09-17T10:01:25.823 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-09-17T10:01:25.930 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-09-17T10:01:25.962 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-09-17T10:01:26.057 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-09-17T10:01:26.100 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-09-17T10:01:26.185 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-09-17T10:01:26.227 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-09-17T10:01:26.333 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-09-17T10:01:26.367 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-09-17T10:01:26.369 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-09-17T10:01:26.473 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-09-17T10:01:26.475 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-09-17T10:01:26.495 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-09-17T10:01:26.603 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-09-17T10:01:26.626 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-09-17T10:01:26.732 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-09-17T10:01:26.765 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-09-17T10:01:26.860 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-09-17T10:01:26.892 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-09-17T10:01:26.894 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-09-17T10:01:27.010 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-09-17T10:01:27.012 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-09-17T10:01:27.034 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-09-17T10:01:27.139 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-09-17T10:01:27.182 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-09-17T10:01:27.278 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-09-17T10:01:27.311 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-09-17T10:01:27.408 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-09-17T10:01:27.451 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-09-17T10:01:27.547 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-09-17T10:01:27.583 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-09-17T10:01:27.700 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-09-17T10:01:27.712 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-09-17T10:01:27.829 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-09-17T10:01:27.851 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-09-17T10:01:27.965 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-09-17T10:01:27.977 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-09-17T10:01:27.979 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-09-17T10:01:28.103 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-09-17T10:01:28.105 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-09-17T10:01:28.115 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-09-17T10:01:28.231 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-09-17T10:01:28.243 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-09-17T10:01:28.359 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-09-17T10:01:28.381 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-09-17T10:01:28.496 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-09-17T10:01:28.508 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-09-17T10:01:28.643 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-09-17T10:01:28.665 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-09-17T10:01:28.772 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-09-17T10:01:28.804 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-09-17T10:01:28.910 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-09-17T10:01:28.936 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-09-17T10:01:28.937 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-17T10:01:29.042 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-09-17T10:01:29.044 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-17T10:01:29.076 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-09-17T10:01:29.171 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-09-17T10:01:29.205 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-09-17T10:01:29.310 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-09-17T10:01:29.332 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-09-17T10:01:29.447 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-09-17T10:01:29.469 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-09-17T10:01:29.574 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-09-17T10:01:29.596 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-09-17T10:01:29.702 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-09-17T10:01:29.734 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-09-17T10:01:29.839 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-09-17T10:01:29.871 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-09-17T10:01:29.987 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-09-17T10:01:29.998 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-09-17T10:01:30.114 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-09-17T10:01:30.137 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-09-17T10:01:30.268 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-09-17T10:01:30.270 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-09-17T10:01:30.395 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-09-17T10:01:30.406 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-09-17T10:01:30.533 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-09-17T10:01:30.535 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-09-17T10:01:30.670 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-09-17T10:01:30.681 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-09-17T10:01:30.799 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-09-17T10:01:30.810 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-09-17T10:01:30.937 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-09-17T10:01:30.948 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-09-17T10:01:31.078 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4 2021-09-17T10:01:31.080 INFO:teuthology.task.ansible.out:d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-09-17T10:01:31.208 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-09-17T10:01:31.220 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-09-17T10:01:31.336 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-09-17T10:01:31.359 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-09-17T10:01:31.477 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-09-17T10:01:31.489 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-09-17T10:01:31.490 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-09-17T10:01:31.618 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-09-17T10:01:31.620 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-09-17T10:01:31.631 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-09-17T10:01:31.743 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-09-17T10:01:31.766 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-09-17T10:01:31.887 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-09-17T10:01:31.895 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-09-17T10:01:32.011 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-09-17T10:01:32.044 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-09-17T10:01:32.161 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-09-17T10:01:32.192 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-09-17T10:01:32.288 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-09-17T10:01:32.343 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-09-17T10:01:32.428 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-09-17T10:01:32.482 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-09-17T10:01:32.578 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-09-17T10:01:32.612 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-09-17T10:01:32.707 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-09-17T10:01:32.750 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-09-17T10:01:32.845 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-09-17T10:01:32.877 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-09-17T10:01:32.971 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-09-17T10:01:33.025 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-09-17T10:01:33.121 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-09-17T10:01:33.153 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-09-17T10:01:33.258 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-09-17T10:01:33.290 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-09-17T10:01:33.386 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-09-17T10:01:33.429 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-09-17T10:01:33.524 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-09-17T10:01:33.566 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-09-17T10:01:33.681 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-09-17T10:01:33.714 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-09-17T10:01:33.716 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-09-17T10:01:33.829 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-09-17T10:01:33.830 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-09-17T10:01:33.840 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-09-17T10:01:33.968 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-09-17T10:01:33.989 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-09-17T10:01:34.118 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-09-17T10:01:34.139 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-09-17T10:01:34.257 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-09-17T10:01:34.279 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-09-17T10:01:34.384 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-09-17T10:01:34.406 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-09-17T10:01:34.526 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-09-17T10:01:34.548 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-09-17T10:01:34.675 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-09-17T10:01:34.697 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-09-17T10:01:34.813 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-09-17T10:01:34.846 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-09-17T10:01:34.942 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-09-17T10:01:34.986 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-09-17T10:01:35.073 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-09-17T10:01:35.139 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-09-17T10:01:35.213 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-09-17T10:01:35.267 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-09-17T10:01:35.341 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-09-17T10:01:35.405 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-09-17T10:01:35.469 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-09-17T10:01:35.533 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-09-17T10:01:35.608 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-09-17T10:01:35.671 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-09-17T10:01:35.735 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-09-17T10:01:35.799 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-09-17T10:01:35.875 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-09-17T10:01:35.937 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-09-17T10:01:36.012 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-09-17T10:01:36.066 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-09-17T10:01:36.139 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-09-17T10:01:36.205 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-09-17T10:01:36.279 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-09-17T10:01:36.344 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-09-17T10:01:36.409 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-09-17T10:01:36.472 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-09-17T10:01:36.546 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-09-17T10:01:36.610 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-09-17T10:01:36.679 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-09-17T10:01:36.751 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-09-17T10:01:36.774 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:36 +0000 (0:00:20.566) 0:00:54.141 ****** 2021-09-17T10:01:36.870 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:36.881 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:36.882 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-09-17T10:01:36.882 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-09-17T10:01:36.896 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-09-17T10:01:36.898 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:36.916 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-09-17T10:01:36.968 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:36.970 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:36.994 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:36 +0000 (0:00:00.220) 0:00:54.361 ****** 2021-09-17T10:01:37.069 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-09-17T10:01:37.081 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.093 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.103 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.115 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.117 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.127 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.138 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.139 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.149 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.160 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.171 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.225 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-09-17T10:01:37.320 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-09-17T10:01:37.331 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.332 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.355 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-09-17T10:01:37.368 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.368 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.380 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.381 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.393 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.404 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.405 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.418 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.429 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.440 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.451 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.463 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.474 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.475 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-09-17T10:01:37.476 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => 2021-09-17T10:01:37.478 INFO:teuthology.task.ansible.out:{"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.489 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.499 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.501 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.511 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.512 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.522 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.524 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.534 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.545 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.556 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.558 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.567 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.569 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.579 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.581 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.590 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.592 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.602 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-09-17T10:01:37.604 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.614 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.616 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 2021-09-17T10:01:37.617 INFO:teuthology.task.ansible.out: 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.627 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.629 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.640 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.641 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.652 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.653 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89b 2021-09-17T10:01:37.655 INFO:teuthology.task.ansible.out:f370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.664 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.666 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.675 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.678 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.688 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.689 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.691 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.702 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.703 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.714 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.716 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.726 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.729 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.738 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.740 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.751 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.753 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.762 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.764 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.775 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.777 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.787 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.789 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.800 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.801 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.812 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.814 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ans 2021-09-17T10:01:37.816 INFO:teuthology.task.ansible.out:ible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.825 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.827 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.838 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.839 INFO:teuthology.task.ansible.out: 2021-09-17T10:01:37.840 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.853 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e 2021-09-17T10:01:37.855 INFO:teuthology.task.ansible.out:73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.864 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.868 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.877 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.879 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.890 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.892 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.907 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.915 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.916 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.927 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.928 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.939 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.942 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.951 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.952 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.952 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.954 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.965 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.966 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.976 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.978 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.990 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:37.991 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.002 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.004 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.014 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.016 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.028 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.039 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.040 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.041 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.041 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.042 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.044 INFO:teuthology.task.ansible.out: skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.055 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.066 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.067 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.067 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.079 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.080 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.093 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.095 INFO:teuthology.task.ansible.out: 2021-09-17T10:01:38.105 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.106 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.117 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.119 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.129 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.131 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.141 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.143 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.143 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"na 2021-09-17T10:01:38.145 INFO:teuthology.task.ansible.out:me": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.154 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.156 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.166 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.168 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.179 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.181 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.191 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.192 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.203 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.205 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.216 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.217 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.227 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.229 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.239 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.241 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.251 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.253 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.263 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.264 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.274 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.277 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.287 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.288 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.299 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.300 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.301 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.301 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": fal 2021-09-17T10:01:38.303 INFO:teuthology.task.ansible.out:se, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.304 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.315 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.316 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.326 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.328 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.340 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.341 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.361 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.363 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.388 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:38 +0000 (0:00:01.393) 0:00:55.755 ****** 2021-09-17T10:01:38.395 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.397 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.398 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.411 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.413 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.422 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.434 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.436 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.446 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.449 INFO:teuthology.task.ansible.out: 2021-09-17T10:01:38.450 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-09-17T10:01:38.450 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.474 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:38 +0000 (0:00:00.086) 0:00:55.842 ****** 2021-09-17T10:01:38.482 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-09-17T10:01:38.483 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.485 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.498 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.508 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.519 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.521 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.530 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.542 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.558 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.564 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.576 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.578 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.589 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.635 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:38 +0000 (0:00:00.160) 0:00:56.002 ****** 2021-09-17T10:01:38.705 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:38.707 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi050.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-09-17T10:01:38.731 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:38 +0000 (0:00:00.096) 0:00:56.098 ****** 2021-09-17T10:01:38.844 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-09-17T10:01:38.960 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-09-17T10:01:38.982 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-09-17T10:01:39.090 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-09-17T10:01:39.126 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-09-17T10:01:39.232 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-09-17T10:01:39.265 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-09-17T10:01:39.372 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-09-17T10:01:39.407 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-09-17T10:01:39.511 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-09-17T10:01:39.543 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-09-17T10:01:39.638 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-09-17T10:01:39.683 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-09-17T10:01:39.779 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-09-17T10:01:39.821 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-09-17T10:01:39.909 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-09-17T10:01:39.964 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-09-17T10:01:40.048 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-09-17T10:01:40.103 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-09-17T10:01:40.188 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-09-17T10:01:40.245 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-09-17T10:01:40.331 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-09-17T10:01:40.385 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-09-17T10:01:40.471 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-09-17T10:01:40.525 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-09-17T10:01:40.600 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-09-17T10:01:40.653 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-09-17T10:01:40.737 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-09-17T10:01:40.793 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-09-17T10:01:40.867 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-09-17T10:01:40.930 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-09-17T10:01:41.004 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-09-17T10:01:41.068 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-09-17T10:01:41.132 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-09-17T10:01:41.205 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-09-17T10:01:41.268 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-09-17T10:01:41.342 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-09-17T10:01:41.406 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-09-17T10:01:41.470 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-09-17T10:01:41.544 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-09-17T10:01:41.620 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-09-17T10:01:41.672 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-09-17T10:01:41.756 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-09-17T10:01:41.810 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-09-17T10:01:41.895 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-09-17T10:01:41.948 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-09-17T10:01:42.032 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-09-17T10:01:42.095 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-09-17T10:01:42.170 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-09-17T10:01:42.223 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-09-17T10:01:42.304 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-09-17T10:01:42.367 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-09-17T10:01:42.443 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-09-17T10:01:42.512 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-09-17T10:01:42.584 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-09-17T10:01:42.648 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-09-17T10:01:42.723 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-09-17T10:01:42.797 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-09-17T10:01:42.863 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-09-17T10:01:42.935 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-09-17T10:01:43.000 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-09-17T10:01:43.073 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-09-17T10:01:43.138 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-09-17T10:01:43.202 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-09-17T10:01:43.276 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-09-17T10:01:43.350 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-09-17T10:01:43.403 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-09-17T10:01:43.487 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-09-17T10:01:43.542 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-09-17T10:01:43.626 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-09-17T10:01:43.680 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-09-17T10:01:43.774 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-09-17T10:01:43.817 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-09-17T10:01:43.902 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-09-17T10:01:43.955 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-09-17T10:01:44.039 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-09-17T10:01:44.176 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-09-17T10:01:44.335 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-09-17T10:01:44.700 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-09-17T10:01:44.754 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-09-17T10:01:45.205 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-09-17T10:01:45.291 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-09-17T10:01:45.713 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-09-17T10:01:45.798 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-09-17T10:01:46.146 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-09-17T10:01:46.233 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-09-17T10:01:46.638 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-09-17T10:01:46.779 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-09-17T10:01:46.926 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-09-17T10:01:47.063 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-09-17T10:01:47.106 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-09-17T10:01:47.201 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-09-17T10:01:47.245 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-09-17T10:01:47.351 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-09-17T10:01:47.382 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-09-17T10:01:47.479 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-09-17T10:01:47.522 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-09-17T10:01:47.617 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-09-17T10:01:47.660 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-09-17T10:01:47.765 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-09-17T10:01:47.798 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-09-17T10:01:47.914 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-09-17T10:01:47.925 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-09-17T10:01:48.042 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-09-17T10:01:48.064 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-09-17T10:01:48.179 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-09-17T10:01:48.212 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-09-17T10:01:48.238 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:48 +0000 (0:00:09.505) 0:01:05.604 ****** 2021-09-17T10:01:48.347 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-09-17T10:01:48.474 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-09-17T10:01:48.538 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:48 +0000 (0:00:00.301) 0:01:05.905 ****** 2021-09-17T10:01:48.963 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "checksum": "4a5787f42b17837e1416d13c4325647102121755", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "c85c4869d3df1b2c6c21c5a0709ecc0c", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1631872908.3285959-20858-241987867750543/source", "state": "file", "uid": 0} 2021-09-17T10:01:48.989 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:48 +0000 (0:00:00.451) 0:01:06.357 ****** 2021-09-17T10:01:48.996 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "checksum": "4a5787f42b17837e1416d13c4325647102121755", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "c85c4869d3df1b2c6c21c5a0709ecc0c", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1631872908.6279624-20936-176368911864149/source", "state": "file", "uid": 0} 2021-09-17T10:01:49.022 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.032) 0:01:06.389 ****** 2021-09-17T10:01:49.059 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-09-17T10:01:49.061 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => { "msg": "Host smithi050.front.sepia.ceph.com is running Ubuntu 20.04 (focal)" } 2021-09-17T10:01:49.085 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.062) 0:01:06.452 ****** 2021-09-17T10:01:49.102 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => { "msg": "Host smithi018.front.sepia.ceph.com is running Ubuntu 20.04 (focal)" } 2021-09-17T10:01:49.130 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.045) 0:01:06.497 ****** 2021-09-17T10:01:49.160 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-09-17T10:01:49.162 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-09-17T10:01:49.190 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.059) 0:01:06.557 ****** 2021-09-17T10:01:49.209 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-09-17T10:01:49.235 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.045) 0:01:06.603 ****** 2021-09-17T10:01:49.264 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-09-17T10:01:49.266 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.289 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.053) 0:01:06.656 ****** 2021-09-17T10:01:49.307 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.332 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.043) 0:01:06.700 ****** 2021-09-17T10:01:49.363 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.391 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.058) 0:01:06.759 ****** 2021-09-17T10:01:49.414 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.443 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.051) 0:01:06.810 ****** 2021-09-17T10:01:49.473 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-09-17T10:01:49.475 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.500 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.056) 0:01:06.867 ****** 2021-09-17T10:01:49.518 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.544 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.044) 0:01:06.911 ****** 2021-09-17T10:01:49.572 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-09-17T10:01:49.574 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.605 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.060) 0:01:06.972 ****** 2021-09-17T10:01:49.614 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.651 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.045) 0:01:07.018 ****** 2021-09-17T10:01:49.670 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-09-17T10:01:49.671 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.696 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.045) 0:01:07.063 ****** 2021-09-17T10:01:49.723 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.750 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.054) 0:01:07.117 ****** 2021-09-17T10:01:49.768 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.796 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.046) 0:01:07.164 ****** 2021-09-17T10:01:49.798 INFO:teuthology.task.ansible.out: 2021-09-17T10:01:49.826 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.851 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.054) 0:01:07.218 ****** 2021-09-17T10:01:49.869 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-09-17T10:01:49.871 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.895 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.043) 0:01:07.262 ****** 2021-09-17T10:01:49.923 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.949 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.054) 0:01:07.316 ****** 2021-09-17T10:01:49.968 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-09-17T10:01:49.969 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:49.992 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:49 +0000 (0:00:00.043) 0:01:07.360 ****** 2021-09-17T10:01:50.020 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.052 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.056) 0:01:07.417 ****** 2021-09-17T10:01:50.078 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-09-17T10:01:50.081 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:01:50.104 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.055) 0:01:07.472 ****** 2021-09-17T10:01:50.142 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:01:50.144 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:01:50.168 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.063) 0:01:07.535 ****** 2021-09-17T10:01:50.175 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-09-17T10:01:50.177 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.203 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.034) 0:01:07.570 ****** 2021-09-17T10:01:50.240 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.265 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.062) 0:01:07.632 ****** 2021-09-17T10:01:50.273 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-09-17T10:01:50.274 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.301 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.035) 0:01:07.668 ****** 2021-09-17T10:01:50.333 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.359 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.058) 0:01:07.726 ****** 2021-09-17T10:01:50.377 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-09-17T10:01:50.379 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.402 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.043) 0:01:07.770 ****** 2021-09-17T10:01:50.431 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.459 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.056) 0:01:07.826 ****** 2021-09-17T10:01:50.483 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-09-17T10:01:50.484 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:01:50.509 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.049) 0:01:07.876 ****** 2021-09-17T10:01:50.538 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:01:50.562 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.053) 0:01:07.930 ****** 2021-09-17T10:01:50.579 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-09-17T10:01:50.581 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.604 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.041) 0:01:07.971 ****** 2021-09-17T10:01:50.632 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.660 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.055) 0:01:08.027 ****** 2021-09-17T10:01:50.677 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-09-17T10:01:50.679 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:01:50.703 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.042) 0:01:08.070 ****** 2021-09-17T10:01:50.730 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:01:50.755 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.052) 0:01:08.123 ****** 2021-09-17T10:01:50.772 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-09-17T10:01:50.774 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.799 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.043) 0:01:08.166 ****** 2021-09-17T10:01:50.817 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.843 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.044) 0:01:08.211 ****** 2021-09-17T10:01:50.871 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-09-17T10:01:50.873 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.898 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.055) 0:01:08.266 ****** 2021-09-17T10:01:50.916 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.943 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.043) 0:01:08.310 ****** 2021-09-17T10:01:50.962 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:50.987 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:50 +0000 (0:00:00.044) 0:01:08.354 ****** 2021-09-17T10:01:51.015 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.041 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.054) 0:01:08.408 ****** 2021-09-17T10:01:51.058 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-09-17T10:01:51.060 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.084 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.042) 0:01:08.451 ****** 2021-09-17T10:01:51.112 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.139 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.053) 0:01:08.505 ****** 2021-09-17T10:01:51.156 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-09-17T10:01:51.158 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:01:51.181 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.043) 0:01:08.548 ****** 2021-09-17T10:01:51.208 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:01:51.238 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.055) 0:01:08.604 ****** 2021-09-17T10:01:51.255 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-17T10:01:51.256 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.282 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.045) 0:01:08.649 ****** 2021-09-17T10:01:51.300 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.326 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.043) 0:01:08.693 ****** 2021-09-17T10:01:51.354 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-09-17T10:01:51.355 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.381 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.055) 0:01:08.748 ****** 2021-09-17T10:01:51.399 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.425 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.044) 0:01:08.793 ****** 2021-09-17T10:01:51.444 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-09-17T10:01:51.446 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.471 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.045) 0:01:08.838 ****** 2021-09-17T10:01:51.489 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.514 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.042) 0:01:08.881 ****** 2021-09-17T10:01:51.541 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-09-17T10:01:51.544 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.569 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.054) 0:01:08.936 ****** 2021-09-17T10:01:51.587 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.613 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.044) 0:01:08.980 ****** 2021-09-17T10:01:51.662 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-09-17T10:01:51.664 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.686 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.073) 0:01:09.054 ****** 2021-09-17T10:01:51.693 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.719 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.032) 0:01:09.086 ****** 2021-09-17T10:01:51.757 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-09-17T10:01:51.759 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.783 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.063) 0:01:09.150 ****** 2021-09-17T10:01:51.801 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.832 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.049) 0:01:09.199 ****** 2021-09-17T10:01:51.860 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-09-17T10:01:51.861 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:01:51.888 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.056) 0:01:09.256 ****** 2021-09-17T10:01:51.906 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:01:51.933 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.044) 0:01:09.301 ****** 2021-09-17T10:01:51.954 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:51.981 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:51 +0000 (0:00:00.047) 0:01:09.348 ****** 2021-09-17T10:01:52.010 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.036 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.055) 0:01:09.403 ****** 2021-09-17T10:01:52.060 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-09-17T10:01:52.062 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.086 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.050) 0:01:09.453 ****** 2021-09-17T10:01:52.114 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.139 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.053) 0:01:09.507 ****** 2021-09-17T10:01:52.158 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-09-17T10:01:52.160 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.185 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.045) 0:01:09.553 ****** 2021-09-17T10:01:52.216 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.243 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.057) 0:01:09.610 ****** 2021-09-17T10:01:52.273 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-09-17T10:01:52.275 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.287 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.311 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.068) 0:01:09.679 ****** 2021-09-17T10:01:52.313 INFO:teuthology.task.ansible.out: 2021-09-17T10:01:52.337 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.339 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.364 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.052) 0:01:09.732 ****** 2021-09-17T10:01:52.404 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-09-17T10:01:52.406 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.434 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.067) 0:01:09.800 ****** 2021-09-17T10:01:52.450 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.452 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.481 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.048) 0:01:09.848 ****** 2021-09-17T10:01:52.521 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-17T10:01:52.523 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.547 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.065) 0:01:09.914 ****** 2021-09-17T10:01:52.554 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.580 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.033) 0:01:09.948 ****** 2021-09-17T10:01:52.755 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-17T10:01:52.757 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.009874", "end": "2021-09-17 10:01:54.032148", "rc": 0, "start": "2021-09-17 10:01:54.022274", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-09-17T10:01:52.781 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.199) 0:01:10.147 ****** 2021-09-17T10:01:52.798 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.015231", "end": "2021-09-17 10:01:52.405017", "rc": 0, "start": "2021-09-17 10:01:52.389786", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-09-17T10:01:52.824 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.043) 0:01:10.191 ****** 2021-09-17T10:01:52.852 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-09-17T10:01:52.853 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.878 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.054) 0:01:10.246 ****** 2021-09-17T10:01:52.896 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.923 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.044) 0:01:10.290 ****** 2021-09-17T10:01:52.951 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-09-17T10:01:52.953 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:52.976 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:52 +0000 (0:00:00.053) 0:01:10.344 ****** 2021-09-17T10:01:52.994 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:53.021 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:53 +0000 (0:00:00.044) 0:01:10.388 ****** 2021-09-17T10:01:53.614 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-09-17T10:01:53.615 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "name": "sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit sleep.target is masked.\"", "LoadState": "masked", "Names": "sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-09-17T10:01:53.641 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "name": "sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit sleep.target is masked.\"", "LoadState": "masked", "Names": "sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-09-17T10:01:53.834 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "name": "suspend.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "suspend.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "suspend.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit suspend.target is masked.\"", "LoadState": "masked", "Names": "suspend.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-09-17T10:01:53.876 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "name": "suspend.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "suspend.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "suspend.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit suspend.target is masked.\"", "LoadState": "masked", "Names": "suspend.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-09-17T10:01:54.025 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "name": "hibernate.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hibernate.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hibernate.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hibernate.target is masked.\"", "LoadState": "masked", "Names": "hibernate.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-09-17T10:01:54.089 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "name": "hibernate.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hibernate.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hibernate.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hibernate.target is masked.\"", "LoadState": "masked", "Names": "hibernate.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-09-17T10:01:54.206 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "name": "hybrid-sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hybrid-sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hybrid-sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hybrid-sleep.target is masked.\"", "LoadState": "masked", "Names": "hybrid-sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-09-17T10:01:54.231 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:54 +0000 (0:00:01.210) 0:01:11.599 ****** 2021-09-17T10:01:54.270 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "name": "hybrid-sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hybrid-sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hybrid-sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hybrid-sleep.target is masked.\"", "LoadState": "masked", "Names": "hybrid-sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-09-17T10:01:54.307 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:54 +0000 (0:00:00.075) 0:01:11.674 ****** 2021-09-17T10:01:54.314 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:54.342 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:54 +0000 (0:00:00.035) 0:01:11.710 ****** 2021-09-17T10:01:54.372 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:54.398 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:54 +0000 (0:00:00.055) 0:01:11.765 ****** 2021-09-17T10:01:54.417 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-09-17T10:01:54.419 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:54.442 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:54 +0000 (0:00:00.044) 0:01:11.810 ****** 2021-09-17T10:01:54.470 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:54.495 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:54 +0000 (0:00:00.052) 0:01:11.863 ****** 2021-09-17T10:01:54.512 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-09-17T10:01:54.514 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:54.538 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:54 +0000 (0:00:00.042) 0:01:11.905 ****** 2021-09-17T10:01:54.567 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:54.592 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:54 +0000 (0:00:00.054) 0:01:11.959 ****** 2021-09-17T10:01:54.640 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-09-17T10:01:54.642 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:54.670 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:54 +0000 (0:00:00.077) 0:01:12.037 ****** 2021-09-17T10:01:54.678 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:54.706 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:54 +0000 (0:00:00.036) 0:01:12.073 ****** 2021-09-17T10:01:54.738 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:54.766 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:54 +0000 (0:00:00.059) 0:01:12.133 ****** 2021-09-17T10:01:54.793 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:54.830 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:54 +0000 (0:00:00.064) 0:01:12.197 ****** 2021-09-17T10:01:55.223 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-09-17T10:01:55.225 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.254223", "end": "2021-09-17 10:01:56.500485", "rc": 0, "start": "2021-09-17 10:01:56.246262", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:01:55.250 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:55 +0000 (0:00:00.419) 0:01:12.617 ****** 2021-09-17T10:01:55.319 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-09-17T10:01:55.321 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:55.344 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:55 +0000 (0:00:00.094) 0:01:12.711 ****** 2021-09-17T10:01:55.623 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.085923", "end": "2021-09-17 10:01:56.892657", "rc": 0, "start": "2021-09-17 10:01:56.806734", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:01:55.646 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:55 +0000 (0:00:00.302) 0:01:13.014 ****** 2021-09-17T10:01:55.888 INFO:teuthology.task.ansible.out: 2021-09-17T10:01:55.890 INFO:teuthology.task.ansible.out:TASK [common : Set the hardware clock] ***************************************** changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.976786", "end": "2021-09-17 10:01:55.500429", "rc": 0, "start": "2021-09-17 10:01:54.523643", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:01:55.916 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:55 +0000 (0:00:00.269) 0:01:13.283 ****** 2021-09-17T10:01:55.998 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:01:56.021 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:56 +0000 (0:00:00.105) 0:01:13.389 ****** 2021-09-17T10:01:56.312 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-09-17T10:01:56.313 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.101488", "end": "2021-09-17 10:01:55.922936", "rc": 0, "start": "2021-09-17 10:01:55.821448", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:01:56.336 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:56 +0000 (0:00:00.314) 0:01:13.703 ****** 2021-09-17T10:01:59.693 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** ok: [smithi050.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:01:59.719 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:01:59 +0000 (0:00:03.381) 0:01:17.085 ****** 2021-09-17T10:02:00.115 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:02:00.140 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:00 +0000 (0:00:00.422) 0:01:17.507 ****** 2021-09-17T10:02:00.462 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-09-17T10:02:00.464 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:02:00.488 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:00 +0000 (0:00:00.347) 0:01:17.855 ****** 2021-09-17T10:02:00.558 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-09-17T10:02:00.561 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:00.585 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:00 +0000 (0:00:00.096) 0:01:17.951 ****** 2021-09-17T10:02:00.884 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:02:00.908 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:00 +0000 (0:00:00.323) 0:01:18.275 ****** 2021-09-17T10:02:00.968 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-09-17T10:02:00.970 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:00.993 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:00 +0000 (0:00:00.085) 0:01:18.360 ****** 2021-09-17T10:02:01.015 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:01 +0000 (0:00:00.021) 0:01:18.382 ****** 2021-09-17T10:02:01.390 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-09-17T10:02:01.412 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:01 +0000 (0:00:00.397) 0:01:18.779 ****** 2021-09-17T10:02:01.745 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-09-17T10:02:01.747 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:02:01.771 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:01 +0000 (0:00:00.358) 0:01:19.138 ****** 2021-09-17T10:02:02.168 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:02:02.201 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:02 +0000 (0:00:00.430) 0:01:19.568 ****** 2021-09-17T10:02:02.208 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-09-17T10:02:02.209 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-09-17T10:02:02.506 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-09-17T10:02:02.626 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-09-17T10:02:02.829 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-09-17T10:02:02.855 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:02 +0000 (0:00:00.654) 0:01:20.223 ****** 2021-09-17T10:02:02.938 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-09-17T10:02:02.991 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-09-17T10:02:02.993 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T10:02:03.017 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:03 +0000 (0:00:00.161) 0:01:20.384 ****** 2021-09-17T10:02:03.248 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-09-17T10:02:03.249 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-09-17T10:02:03.274 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:03 +0000 (0:00:00.257) 0:01:20.642 ****** 2021-09-17T10:02:03.402 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi050.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-09-17T10:02:03.424 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-09-17T10:02:03.426 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T10:02:03.450 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:03 +0000 (0:00:00.175) 0:01:20.817 ****** 2021-09-17T10:02:03.701 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-17T10:02:03.702 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-09-17T10:02:03.842 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-09-17T10:02:04.026 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3612, "state": "file", "uid": 0} 2021-09-17T10:02:04.058 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:04 +0000 (0:00:00.607) 0:01:21.425 ****** 2021-09-17T10:02:04.150 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-09-17T10:02:04.152 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-09-17T10:02:04.153 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-09-17T10:02:04.176 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:04 +0000 (0:00:00.118) 0:01:21.544 ****** 2021-09-17T10:02:04.247 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:04.270 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:04 +0000 (0:00:00.093) 0:01:21.638 ****** 2021-09-17T10:02:04.340 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-09-17T10:02:04.342 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:02:04.367 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:04 +0000 (0:00:00.096) 0:01:21.734 ****** 2021-09-17T10:02:04.458 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-09-17T10:02:04.460 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:04.470 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:04.481 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-17T10:02:04.482 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3612, "state": "file", "uid": 0} TASK [common : Install nrpe without epel] ************************************** skipping: [smithi050.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:04.483 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-17T10:02:04.510 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:04 +0000 (0:00:00.142) 0:01:21.877 ****** 2021-09-17T10:02:04.520 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-09-17T10:02:04.522 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:04.556 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:04 +0000 (0:00:00.045) 0:01:21.923 ****** 2021-09-17T10:02:04.584 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-09-17T10:02:04.586 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-09-17T10:02:04.610 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:04 +0000 (0:00:00.054) 0:01:21.977 ****** 2021-09-17T10:02:04.627 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:04.654 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:04 +0000 (0:00:00.042) 0:01:22.020 ****** 2021-09-17T10:02:04.680 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:04.706 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:04 +0000 (0:00:00.053) 0:01:22.073 ****** 2021-09-17T10:02:04.786 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* skipping: [smithi018.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:02:04.814 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:04 +0000 (0:00:00.107) 0:01:22.180 ****** 2021-09-17T10:02:04.905 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-09-17T10:02:04.907 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:04.927 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:04.937 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:04.952 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:04.976 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:04 +0000 (0:00:00.162) 0:01:22.343 ****** 2021-09-17T10:02:05.055 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-09-17T10:02:05.057 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:05.081 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:05 +0000 (0:00:00.105) 0:01:22.449 ****** 2021-09-17T10:02:05.416 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:02:05.440 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:05 +0000 (0:00:00.358) 0:01:22.807 ****** 2021-09-17T10:02:05.832 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-09-17T10:02:05.835 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-09-17T10:02:05.856 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:05 +0000 (0:00:00.415) 0:01:23.223 ****** 2021-09-17T10:02:05.894 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:02:05.920 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:05 +0000 (0:00:00.063) 0:01:23.287 ****** 2021-09-17T10:02:06.262 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-09-17T10:02:06.264 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-09-17T10:02:06.287 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:06 +0000 (0:00:00.367) 0:01:23.654 ****** 2021-09-17T10:02:06.304 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-09-17T10:02:06.306 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-09-17T10:02:06.333 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:06 +0000 (0:00:00.046) 0:01:23.701 ****** 2021-09-17T10:02:06.488 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-09-17T10:02:06.490 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-09-17T10:02:06.616 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-09-17T10:02:06.642 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:06 +0000 (0:00:00.308) 0:01:24.009 ****** 2021-09-17T10:02:06.711 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:06.736 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:06 +0000 (0:00:00.094) 0:01:24.103 ****** 2021-09-17T10:02:06.743 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-09-17T10:02:06.746 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-09-17T10:02:06.770 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:06 +0000 (0:00:00.033) 0:01:24.137 ****** 2021-09-17T10:02:06.808 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-09-17T10:02:06.810 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:02:06.833 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:06 +0000 (0:00:00.063) 0:01:24.200 ****** 2021-09-17T10:02:06.913 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-09-17T10:02:06.915 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:06.937 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:06 +0000 (0:00:00.103) 0:01:24.304 ****** 2021-09-17T10:02:06.995 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-09-17T10:02:06.997 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-09-17T10:02:07.123 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-09-17T10:02:07.153 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:07 +0000 (0:00:00.216) 0:01:24.520 ****** 2021-09-17T10:02:07.226 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:07.250 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:07 +0000 (0:00:00.096) 0:01:24.617 ****** 2021-09-17T10:02:07.319 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi018.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-17T10:02:07.343 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:07 +0000 (0:00:00.093) 0:01:24.710 ****** 2021-09-17T10:02:07.351 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-09-17T10:02:07.353 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-09-17T10:02:07.383 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:07 +0000 (0:00:00.039) 0:01:24.750 ****** 2021-09-17T10:02:07.444 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-09-17T10:02:07.446 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:07.469 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:07 +0000 (0:00:00.086) 0:01:24.837 ****** 2021-09-17T10:02:07.656 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-09-17T10:02:07.658 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-09-17 10:00:11 UTC", "ActiveEnterTimestampMonotonic": "16833041", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "local-fs.target sysinit.target nss-lookup.target network.target var-run.mount systemd-tmpfiles-setup.service -.mount systemd-journald.socket system.slice remote-fs.target basic.target time-sync.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-09-17 10:00:11 UTC", "AssertTimestampMonotonic": "16824662", "Before": "xdm.service shutdown.target getty@tty1.service plymouth-quit.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-09-17 10:00:11 UTC", "Co 2021-09-17T10:02:07.659 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "16824662", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "840", "ExecMainStartTimestamp": "Fri 2021-09-17 10:00:11 UTC", "ExecMainStartTimestampMonotonic": "16832910", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Fri 2021-09-17 10:00:11 UTC] ; stop_time=[n/a] ; pid=840 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; flags= ; start_time=[Fri 2021-09-17 10:00:11 UTC] ; stop_time=[n/a] ; pid=840 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPostEx": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GI 2021-09-17T10:02:07.661 INFO:teuthology.task.ansible.out:D": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-09-17 10:00:11 UTC", "InactiveExitTimestampMonotonic": "16833041", "InvocationID": "86939c2ff6c14c2e8d3802f88acd8891", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127802", "LimitNPROCSoft": "127802", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127802", "LimitSIGPENDINGSoft": "127802", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "840", "MemoryAccounting": "yes", "MemoryCurrent": "1449984", "MemoryD 2021-09-17T10:02:07.663 INFO:teuthology.task.ansible.out:enyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nagios/nrpe.pid", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-09-17 10:00:11 UTC", "StateChangeTimestampMonotonic": "16833041", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no" 2021-09-17T10:02:07.664 INFO:teuthology.task.ansible.out:, "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "38340", "TimeoutAbortUSec": "1min", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "110", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-17T10:02:07.683 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:07 +0000 (0:00:00.213) 0:01:25.051 ****** 2021-09-17T10:02:07.752 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-09-17T10:02:07.754 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:07.777 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:07 +0000 (0:00:00.093) 0:01:25.144 ****** 2021-09-17T10:02:07.845 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-09-17T10:02:07.847 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:07.876 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:07 +0000 (0:00:00.098) 0:01:25.243 ****** 2021-09-17T10:02:07.883 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-09-17T10:02:07.885 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-09-17T10:02:07.916 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:07 +0000 (0:00:00.039) 0:01:25.283 ****** 2021-09-17T10:02:07.947 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-09-17T10:02:07.949 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:07.972 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:07 +0000 (0:00:00.056) 0:01:25.340 ****** 2021-09-17T10:02:08.042 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-09-17T10:02:08.043 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.073 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.097) 0:01:25.437 ****** 2021-09-17T10:02:08.170 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-09-17T10:02:08.172 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.195 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.125) 0:01:25.562 ****** 2021-09-17T10:02:08.204 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-09-17T10:02:08.206 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-09-17 10:00:04 UTC", "ActiveEnterTimestampMonotonic": "15740027", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service nss-lookup.target basic.target remote-fs.target systemd-journald.socket local-fs.target sysinit.target var-run.mount network.target time-sync.target system.slice -.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-09-17 10:00:04 UTC", "AssertTimestampMonotonic": "15731626", "Before": "xdm.service shutdown.target getty@tty1.service multi-user.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-09-17 10:00:04 UTC", "Co 2021-09-17T10:02:08.207 INFO:teuthology.task.ansible.out:nditionTimestampMonotonic": "15731625", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "841", "ExecMainStartTimestamp": "Fri 2021-09-17 10:00:04 UTC", "ExecMainStartTimestampMonotonic": "15739880", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Fri 2021-09-17 10:00:04 UTC] ; stop_time=[n/a] ; pid=841 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; flags= ; start_time=[Fri 2021-09-17 10:00:04 UTC] ; stop_time=[n/a] ; pid=841 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPostEx": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GI 2021-09-17T10:02:08.209 INFO:teuthology.task.ansible.out:D": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-09-17 10:00:04 UTC", "InactiveExitTimestampMonotonic": "15740027", "InvocationID": "b58e379adf714b658cf296107e120556", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127802", "LimitNPROCSoft": "127802", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127802", "LimitSIGPENDINGSoft": "127802", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "841", "MemoryAccounting": "yes", "MemoryCurrent": "1368064", "MemoryD 2021-09-17T10:02:08.210 INFO:teuthology.task.ansible.out:enyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nagios/nrpe.pid", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-09-17 10:00:04 UTC", "StateChangeTimestampMonotonic": "15740027", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no" 2021-09-17T10:02:08.212 INFO:teuthology.task.ansible.out:, "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "38340", "TimeoutAbortUSec": "1min", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "110", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-17T10:02:08.235 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.040) 0:01:25.603 ****** 2021-09-17T10:02:08.263 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-09-17T10:02:08.265 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.290 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.052) 0:01:25.656 ****** 2021-09-17T10:02:08.317 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-09-17T10:02:08.319 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.347 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.058) 0:01:25.715 ****** 2021-09-17T10:02:08.368 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-09-17T10:02:08.370 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.396 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.048) 0:01:25.763 ****** 2021-09-17T10:02:08.425 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-09-17T10:02:08.427 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.455 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.059) 0:01:25.822 ****** 2021-09-17T10:02:08.492 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-09-17T10:02:08.494 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.532 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.075) 0:01:25.898 ****** 2021-09-17T10:02:08.560 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-09-17T10:02:08.562 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.589 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.057) 0:01:25.956 ****** 2021-09-17T10:02:08.606 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-09-17T10:02:08.608 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.634 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.045) 0:01:26.002 ****** 2021-09-17T10:02:08.663 INFO:teuthology.task.ansible.out: 2021-09-17T10:02:08.664 INFO:teuthology.task.ansible.out:TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-09-17T10:02:08.664 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.696 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.061) 0:01:26.063 ****** 2021-09-17T10:02:08.719 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.759 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.061) 0:01:26.125 ****** 2021-09-17T10:02:08.799 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-09-17T10:02:08.801 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.829 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.071) 0:01:26.196 ****** 2021-09-17T10:02:08.850 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-09-17T10:02:08.886 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.057) 0:01:26.254 ****** 2021-09-17T10:02:08.919 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-09-17T10:02:08.920 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:08.956 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:08 +0000 (0:00:00.069) 0:01:26.323 ****** 2021-09-17T10:02:08.988 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-09-17T10:02:08.989 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-09-17T10:02:09.022 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.066) 0:01:26.390 ****** 2021-09-17T10:02:09.025 INFO:teuthology.task.ansible.out: 2021-09-17T10:02:09.042 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-09-17T10:02:09.044 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:09.070 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.047) 0:01:26.437 ****** 2021-09-17T10:02:09.098 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-09-17T10:02:09.100 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "docker.io", "python3-nose", "python3-venv", "python3-virtualenv", "python3-configobj", "python3-gevent", "python3-numpy", "python3-matplotlib", "python3-setuptools"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"} 2021-09-17T10:02:09.126 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.055) 0:01:26.493 ****** 2021-09-17T10:02:09.146 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-09-17T10:02:09.148 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:09.174 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.048) 0:01:26.542 ****** 2021-09-17T10:02:09.212 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-09-17T10:02:09.214 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-17T10:02:09.253 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.077) 0:01:26.619 ****** 2021-09-17T10:02:09.258 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-09-17T10:02:09.259 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:09.285 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.033) 0:01:26.652 ****** 2021-09-17T10:02:09.379 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-09-17T10:02:09.380 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:09.403 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.117) 0:01:26.770 ****** 2021-09-17T10:02:09.411 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-09-17T10:02:09.413 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-09-17T10:02:09.439 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.036) 0:01:26.806 ****** 2021-09-17T10:02:09.490 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-09-17T10:02:09.492 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-09-17T10:02:09.524 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.084) 0:01:26.891 ****** 2021-09-17T10:02:09.599 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-09-17T10:02:09.601 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-09-17T10:02:09.639 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.115) 0:01:27.007 ****** 2021-09-17T10:02:09.647 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-09-17T10:02:09.648 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-09-17T10:02:09.674 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.034) 0:01:27.041 ****** 2021-09-17T10:02:09.715 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-09-17T10:02:09.717 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "docker.io", "python3-nose", "python3-venv", "python3-virtualenv", "python3-configobj", "python3-gevent", "python3-numpy", "python3-matplotlib", "python3-setuptools"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"} 2021-09-17T10:02:09.742 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.068) 0:01:27.110 ****** 2021-09-17T10:02:09.750 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-09-17T10:02:09.752 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:09.778 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.036) 0:01:27.146 ****** 2021-09-17T10:02:09.817 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-09-17T10:02:09.819 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-17T10:02:09.849 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.070) 0:01:27.216 ****** 2021-09-17T10:02:09.856 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-09-17T10:02:09.857 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:09.885 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:09 +0000 (0:00:00.036) 0:01:27.253 ****** 2021-09-17T10:02:09.989 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-09-17T10:02:09.991 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-09-17T10:02:10.017 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:10 +0000 (0:00:00.129) 0:01:27.382 ****** 2021-09-17T10:02:10.106 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-09-17T10:02:10.108 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-09-17T10:02:10.132 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:10 +0000 (0:00:00.117) 0:01:27.500 ****** 2021-09-17T10:02:10.204 INFO:teuthology.task.ansible.out: 2021-09-17T10:02:10.205 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-09-17T10:02:10.207 INFO:teuthology.task.ansible.out: 2021-09-17T10:02:10.232 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:10 +0000 (0:00:00.099) 0:01:27.599 ****** 2021-09-17T10:02:10.303 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:10.329 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:10 +0000 (0:00:00.096) 0:01:27.696 ****** 2021-09-17T10:02:10.404 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-09-17T10:02:10.406 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:10.429 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:10 +0000 (0:00:00.100) 0:01:27.796 ****** 2021-09-17T10:02:10.612 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-09-17T10:02:10.654 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:10 +0000 (0:00:00.225) 0:01:28.021 ****** 2021-09-17T10:02:10.665 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi018.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-09-17T10:02:10.698 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:10 +0000 (0:00:00.044) 0:01:28.066 ****** 2021-09-17T10:02:10.732 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi050"}, "changed": false} 2021-09-17T10:02:10.757 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:10 +0000 (0:00:00.058) 0:01:28.124 ****** 2021-09-17T10:02:11.167 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-09-17T10:02:11.169 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-09-17T10:02:11.192 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:11 +0000 (0:00:00.434) 0:01:28.559 ****** 2021-09-17T10:02:11.263 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-09-17T10:02:11.264 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi018"}, "changed": false} 2021-09-17T10:02:11.288 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:11 +0000 (0:00:00.095) 0:01:28.655 ****** 2021-09-17T10:02:11.649 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi050.front.sepia.ceph.com", "ansible_hostname": "smithi050", "ansible_nodename": "smithi050"}, "changed": true, "name": "smithi050"} 2021-09-17T10:02:11.674 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:11 +0000 (0:00:00.385) 0:01:29.041 ****** 2021-09-17T10:02:11.756 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi018.front.sepia.ceph.com", "ansible_hostname": "smithi018", "ansible_nodename": "smithi018"}, "changed": true, "name": "smithi018"} 2021-09-17T10:02:11.780 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:11 +0000 (0:00:00.106) 0:01:29.148 ****** 2021-09-17T10:02:12.062 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-09-17T10:02:12.087 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:12 +0000 (0:00:00.306) 0:01:29.454 ****** 2021-09-17T10:02:12.173 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-09-17T10:02:12.201 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:12 +0000 (0:00:00.114) 0:01:29.568 ****** 2021-09-17T10:02:12.501 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-09-17T10:02:12.527 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:12 +0000 (0:00:00.325) 0:01:29.894 ****** 2021-09-17T10:02:12.599 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-09-17T10:02:12.624 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:12 +0000 (0:00:00.097) 0:01:29.992 ****** 2021-09-17T10:02:13.074 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi050.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-09-17T10:02:13.101 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.476) 0:01:30.468 ****** 2021-09-17T10:02:13.120 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-09-17T10:02:13.148 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.046) 0:01:30.515 ****** 2021-09-17T10:02:13.168 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-09-17T10:02:13.170 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.195 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.047) 0:01:30.562 ****** 2021-09-17T10:02:13.223 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.249 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.054) 0:01:30.616 ****** 2021-09-17T10:02:13.266 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-17T10:02:13.268 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.295 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.043) 0:01:30.660 ****** 2021-09-17T10:02:13.322 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.352 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.059) 0:01:30.719 ****** 2021-09-17T10:02:13.370 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-17T10:02:13.372 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.398 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.046) 0:01:30.765 ****** 2021-09-17T10:02:13.427 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.456 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.057) 0:01:30.823 ****** 2021-09-17T10:02:13.473 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-09-17T10:02:13.475 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.500 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.044) 0:01:30.867 ****** 2021-09-17T10:02:13.530 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.562 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.060) 0:01:30.928 ****** 2021-09-17T10:02:13.578 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-17T10:02:13.580 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.605 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.044) 0:01:30.972 ****** 2021-09-17T10:02:13.623 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.649 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.044) 0:01:31.016 ****** 2021-09-17T10:02:13.669 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-17T10:02:13.672 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.696 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.047) 0:01:31.063 ****** 2021-09-17T10:02:13.724 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.751 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.054) 0:01:31.118 ****** 2021-09-17T10:02:13.769 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-09-17T10:02:13.771 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.797 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.045) 0:01:31.164 ****** 2021-09-17T10:02:13.826 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.851 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.053) 0:01:31.218 ****** 2021-09-17T10:02:13.870 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-09-17T10:02:13.872 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.897 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.046) 0:01:31.264 ****** 2021-09-17T10:02:13.927 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:13.952 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:13 +0000 (0:00:00.055) 0:01:31.320 ****** 2021-09-17T10:02:13.983 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T10:02:13.985 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.010 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.057) 0:01:31.378 ****** 2021-09-17T10:02:14.030 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.060 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.049) 0:01:31.427 ****** 2021-09-17T10:02:14.106 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* skipping: [smithi050.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.133 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.073) 0:01:31.501 ****** 2021-09-17T10:02:14.152 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.180 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.046) 0:01:31.547 ****** 2021-09-17T10:02:14.211 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-09-17T10:02:14.212 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.239 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.058) 0:01:31.605 ****** 2021-09-17T10:02:14.258 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.285 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.046) 0:01:31.652 ****** 2021-09-17T10:02:14.302 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-09-17T10:02:14.329 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.044) 0:01:31.696 ****** 2021-09-17T10:02:14.372 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.042) 0:01:31.739 ****** 2021-09-17T10:02:14.391 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-09-17T10:02:14.417 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.044) 0:01:31.784 ****** 2021-09-17T10:02:14.470 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.053) 0:01:31.837 ****** 2021-09-17T10:02:14.498 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-09-17T10:02:14.526 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.056) 0:01:31.893 ****** 2021-09-17T10:02:14.568 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.042) 0:01:31.935 ****** 2021-09-17T10:02:14.596 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-09-17T10:02:14.598 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.622 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.054) 0:01:31.990 ****** 2021-09-17T10:02:14.651 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.682 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.059) 0:01:32.049 ****** 2021-09-17T10:02:14.700 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-09-17T10:02:14.702 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.728 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.046) 0:01:32.096 ****** 2021-09-17T10:02:14.756 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.760 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.786 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.057) 0:01:32.153 ****** 2021-09-17T10:02:14.805 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.831 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.044) 0:01:32.198 ****** 2021-09-17T10:02:14.849 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.876 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.044) 0:01:32.242 ****** 2021-09-17T10:02:14.892 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T10:02:14.894 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.921 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.045) 0:01:32.288 ****** 2021-09-17T10:02:14.938 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:14.967 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:14 +0000 (0:00:00.046) 0:01:32.334 ****** 2021-09-17T10:02:15.005 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-09-17T10:02:15.007 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.031 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.064) 0:01:32.399 ****** 2021-09-17T10:02:15.050 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.077 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.045) 0:01:32.444 ****** 2021-09-17T10:02:15.157 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-09-17T10:02:15.159 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.183 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.105) 0:01:32.550 ****** 2021-09-17T10:02:15.243 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.267 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.084) 0:01:32.635 ****** 2021-09-17T10:02:15.274 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-09-17T10:02:15.277 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.301 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.033) 0:01:32.668 ****** 2021-09-17T10:02:15.340 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.371 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.070) 0:01:32.738 ****** 2021-09-17T10:02:15.378 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-09-17T10:02:15.380 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.413 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.041) 0:01:32.780 ****** 2021-09-17T10:02:15.444 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.470 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.057) 0:01:32.837 ****** 2021-09-17T10:02:15.497 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-17T10:02:15.499 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.528 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.057) 0:01:32.895 ****** 2021-09-17T10:02:15.546 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.572 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.044) 0:01:32.939 ****** 2021-09-17T10:02:15.589 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-17T10:02:15.591 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.620 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.046) 0:01:32.986 ****** 2021-09-17T10:02:15.669 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.696 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.077) 0:01:33.063 ****** 2021-09-17T10:02:15.702 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-17T10:02:15.704 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.729 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.033) 0:01:33.097 ****** 2021-09-17T10:02:15.768 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.800 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.070) 0:01:33.168 ****** 2021-09-17T10:02:15.947 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-09-17T10:02:15.949 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:15.980 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:15 +0000 (0:00:00.179) 0:01:33.348 ****** 2021-09-17T10:02:16.034 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:16.069 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:16 +0000 (0:00:00.087) 0:01:33.435 ****** 2021-09-17T10:02:16.216 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-17T10:02:16.218 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:16.242 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:16 +0000 (0:00:00.174) 0:01:33.610 ****** 2021-09-17T10:02:16.325 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:16.358 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:16 +0000 (0:00:00.115) 0:01:33.725 ****** 2021-09-17T10:02:16.471 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-17T10:02:16.473 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:16.636 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:16 +0000 (0:00:00.277) 0:01:34.003 ****** 2021-09-17T10:02:16.644 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:16.672 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:16 +0000 (0:00:00.035) 0:01:34.039 ****** 2021-09-17T10:02:16.722 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-09-17T10:02:16.723 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:16.750 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:16 +0000 (0:00:00.077) 0:01:34.117 ****** 2021-09-17T10:02:16.757 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:16.787 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:16 +0000 (0:00:00.035) 0:01:34.152 ****** 2021-09-17T10:02:16.814 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-09-17T10:02:16.817 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:16.840 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:16 +0000 (0:00:00.054) 0:01:34.207 ****** 2021-09-17T10:02:16.859 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:16.884 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:16 +0000 (0:00:00.044) 0:01:34.252 ****** 2021-09-17T10:02:16.912 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-09-17T10:02:16.914 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:16.938 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:16 +0000 (0:00:00.053) 0:01:34.305 ****** 2021-09-17T10:02:16.956 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:16.982 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:16 +0000 (0:00:00.044) 0:01:34.349 ****** 2021-09-17T10:02:17.011 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-09-17T10:02:17.013 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:17.039 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.055) 0:01:34.405 ****** 2021-09-17T10:02:17.056 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:17.084 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.046) 0:01:34.452 ****** 2021-09-17T10:02:17.116 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-09-17T10:02:17.118 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:17.143 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.058) 0:01:34.510 ****** 2021-09-17T10:02:17.182 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:17.210 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.066) 0:01:34.577 ****** 2021-09-17T10:02:17.227 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-09-17T10:02:17.231 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:17.257 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.047) 0:01:34.624 ****** 2021-09-17T10:02:17.286 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:17.314 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.056) 0:01:34.681 ****** 2021-09-17T10:02:17.332 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-09-17T10:02:17.334 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:17.360 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.046) 0:01:34.727 ****** 2021-09-17T10:02:17.378 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:17.405 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.045) 0:01:34.772 ****** 2021-09-17T10:02:17.433 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:17.459 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.053) 0:01:34.826 ****** 2021-09-17T10:02:17.477 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:17.504 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.044) 0:01:34.871 ****** 2021-09-17T10:02:17.647 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-09-17T10:02:17.651 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005314", "end": "2021-09-17 10:02:17.267755", "rc": 0, "start": "2021-09-17 10:02:17.262441", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:02:17.674 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.169) 0:01:35.041 ****** 2021-09-17T10:02:17.703 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005289", "end": "2021-09-17 10:02:18.978010", "rc": 0, "start": "2021-09-17 10:02:18.972721", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:02:17.730 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.055) 0:01:35.097 ****** 2021-09-17T10:02:17.747 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-09-17T10:02:17.776 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.045) 0:01:35.143 ****** 2021-09-17T10:02:17.832 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:17 +0000 (0:00:00.056) 0:01:35.199 ****** 2021-09-17T10:02:18.193 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-09-17T10:02:18.194 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-09-17T10:02:18.225 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:18 +0000 (0:00:00.393) 0:01:35.593 ****** 2021-09-17T10:02:18.233 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-09-17T10:02:18.269 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:18 +0000 (0:00:00.043) 0:01:35.636 ****** 2021-09-17T10:02:18.312 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:18.340 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:18 +0000 (0:00:00.071) 0:01:35.707 ****** 2021-09-17T10:02:18.347 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:18.387 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:18 +0000 (0:00:00.047) 0:01:35.754 ****** 2021-09-17T10:02:19.636 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* ok: [smithi018.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-09-17T10:02:19.678 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-09-17T10:02:20.726 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-09-17T10:02:20.752 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:20 +0000 (0:00:02.364) 0:01:38.119 ****** 2021-09-17T10:02:20.791 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-09-17T10:02:20.817 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:20 +0000 (0:00:00.065) 0:01:38.184 ****** 2021-09-17T10:02:21.501 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:02:21.524 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:21 +0000 (0:00:00.707) 0:01:38.892 ****** 2021-09-17T10:02:21.564 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:02:21.589 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:21 +0000 (0:00:00.064) 0:01:38.957 ****** 2021-09-17T10:02:21.606 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-09-17T10:02:21.633 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:21 +0000 (0:00:00.043) 0:01:39.000 ****** 2021-09-17T10:02:21.686 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:21 +0000 (0:00:00.053) 0:01:39.053 ****** 2021-09-17T10:02:21.861 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-09-17T10:02:21.863 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.032184", "end": "2021-09-17 10:02:21.473139", "rc": 0, "start": "2021-09-17 10:02:21.440955", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:02:21.888 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:21 +0000 (0:00:00.201) 0:01:39.255 ****** 2021-09-17T10:02:21.908 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.031533", "end": "2021-09-17 10:02:23.178511", "rc": 0, "start": "2021-09-17 10:02:23.146978", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:02:21.933 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:21 +0000 (0:00:00.045) 0:01:39.301 ****** 2021-09-17T10:02:24.185 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-09-17T10:02:24.187 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:02:24.209 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:24 +0000 (0:00:02.275) 0:01:41.576 ****** 2021-09-17T10:02:24.239 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:02:24.265 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:24 +0000 (0:00:00.055) 0:01:41.632 ****** 2021-09-17T10:02:32.110 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-09-17T10:02:32.112 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1\n libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0\n libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0\n libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0\n libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla\n libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0\n libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf\n qemu-system-data qemu-system-gui seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 175 not upgraded.\nAfter this operation, 62.7 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138110 files and directories currently installed.)\r 2021-09-17T10:02:32.114 INFO:teuthology.task.ansible.out:\nRemoving qemu-utils (1:4.2-3ubuntu6.17) ...\r\nRemoving qemu-system-x86 (1:4.2-3ubuntu6.17) ...\r\nRemoving qemu-system-common (1:4.2-3ubuntu6.17) ...\r\nRemoving qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...\r\nRemoving librbd1 (15.2.13-0ubuntu0.20.04.1) ...\r\nRemoving librados2 (15.2.13-0ubuntu0.20.04.1) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1", " libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0", " libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0", " libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0", " libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla", " libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0", " libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf", " qemu-system-data qemu-system-gui seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 175 not upgraded.", "After this operation, 62.7 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "( 2021-09-17T10:02:32.116 INFO:teuthology.task.ansible.out:Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138110 files and directories currently installed.)", "Removing qemu-utils (1:4.2-3ubuntu6.17) ...", "Removing qemu-system-x86 (1:4.2-3ubuntu6.17) ...", "Removing qemu-system-common (1:4.2-3ubuntu6.17) ...", "Removing qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...", "Removing librbd1 (15.2.13-0ubuntu0.20.04.1) ...", "Removing librados2 (15.2.13-0ubuntu0.20.04.1) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-09-17T10:02:32.139 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:32 +0000 (0:00:07.874) 0:01:49.507 ****** 2021-09-17T10:02:32.209 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1\n libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0\n libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0\n libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0\n libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla\n libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0\n libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf\n qemu-system-data qemu-system-gui seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 175 not upgraded.\nAfter this operation, 62.7 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138110 files and directories currently installed.)\r 2021-09-17T10:02:32.210 INFO:teuthology.task.ansible.out:\nRemoving qemu-utils (1:4.2-3ubuntu6.17) ...\r\nRemoving qemu-system-x86 (1:4.2-3ubuntu6.17) ...\r\nRemoving qemu-system-common (1:4.2-3ubuntu6.17) ...\r\nRemoving qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...\r\nRemoving librbd1 (15.2.13-0ubuntu0.20.04.1) ...\r\nRemoving librados2 (15.2.13-0ubuntu0.20.04.1) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1", " libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0", " libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0", " libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0", " libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla", " libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0", " libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf", " qemu-system-data qemu-system-gui seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 175 not upgraded.", "After this operation, 62.7 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "( 2021-09-17T10:02:32.212 INFO:teuthology.task.ansible.out:Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138110 files and directories currently installed.)", "Removing qemu-utils (1:4.2-3ubuntu6.17) ...", "Removing qemu-system-x86 (1:4.2-3ubuntu6.17) ...", "Removing qemu-system-common (1:4.2-3ubuntu6.17) ...", "Removing qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...", "Removing librbd1 (15.2.13-0ubuntu0.20.04.1) ...", "Removing librados2 (15.2.13-0ubuntu0.20.04.1) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-09-17T10:02:32.237 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:32 +0000 (0:00:00.096) 0:01:49.603 ****** 2021-09-17T10:02:32.865 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-17T10:02:32.867 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:02:32.891 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:32 +0000 (0:00:00.655) 0:01:50.258 ****** 2021-09-17T10:02:32.981 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-09-17T10:02:32.983 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Upgrade packages] ********************************************* ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:02:33.005 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:33 +0000 (0:00:00.114) 0:01:50.373 ****** 2021-09-17T10:02:33.028 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:33 +0000 (0:00:00.022) 0:01:50.396 ****** 2021-09-17T10:02:33.077 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-09-17T10:02:33.079 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:33.103 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:33 +0000 (0:00:00.074) 0:01:50.470 ****** 2021-09-17T10:02:55.034 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-17T10:02:55.036 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n libpython3.8 libpython3.8-dev libpython3.8-minimal libpython3.8-stdlib\n librados2 librbd1 python3.8 python3.8-dev python3.8-minimal python3.8-venv\n qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n gitweb git-cvs git-mediawiki git-svn python3.8-doc binfmt-support samba vde2\n debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 python3-venv python3.8-venv qemu-block-extra\n qemu-system-common qemu-system-x86 qemu-utils\nThe following packages will be upgraded:\n git libpython3.8 libpython3.8-dev libpython3.8-minimal libpython3.8-stdlib\n python3.8 python3.8-dev python3.8-minimal\n8 upgraded, 8 newly installed, 0 to remove and 167 not upgraded.\nNeed to get 29.0 MB of archives.\nAfter this operation, 62.8 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-dev amd64 3.8.10-0ubuntu1~20.04 [510 kB]\nGet:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8 amd64 3.8.10-0ubuntu1~20.04 [387 kB]\nGet:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-dev amd64 3.8.10-0ubuntu1~20.04 [3943 kB]\nGet:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8 amd64 3.8.10-0ubuntu1~20.04 [1625 kB]\nGet:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-stdlib amd64 3.8.10-0ubuntu1~20.04 [1675 kB]\nGet:6 http://archive.ubuntu.com/ubuntu focal-u 2021-09-17T10:02:55.038 INFO:teuthology.task.ansible.out:pdates/main amd64 python3.8-minimal amd64 3.8.10-0ubuntu1~20.04 [1898 kB]\nGet:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-minimal amd64 3.8.10-0ubuntu1~20.04 [717 kB]\nGet:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.2 [4554 kB]\nGet:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.13-0ubuntu0.20.04.2 [3210 kB]\nGet:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.13-0ubuntu0.20.04.2 [1624 kB]\nGet:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3.8-venv amd64 3.8.10-0ubuntu1~20.04 [5456 B]\nGet:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-venv amd64 3.8.2-0ubuntu2 [1228 B]\nGet:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.17 [54.4 kB]\nGet:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.17 [1055 kB]\nGet:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.17 [6738 kB]\nGet:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.17 [973 kB]\nFetched 29.0 MB in 2s (13.6 MB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138038 files and directories currently installed.)\r\nPreparing to unpack .../00-python3.8-dev_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking python3.8-de 2021-09-17T10:02:55.040 INFO:teuthology.task.ansible.out:v (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../01-python3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking python3.8 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../02-libpython3.8-dev_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../03-libpython3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking libpython3.8:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../04-libpython3.8-stdlib_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../05-python3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking python3.8-minimal (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../06-libpython3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../07-git_1%3a2.25.1-1ubuntu3.2_amd64.deb ...\r\nUnpacking git (1:2.25.1-1ubuntu3.2) over (1:2.25.1-1ubuntu3.1) ...\r\nSelecting previously unselected package librados2.\r\nPreparing to unpack .../08-librados2_15.2.13-0ubuntu0.20.04.2_amd64.deb ...\r\nUnpacking librados2 (15.2.13-0ubuntu0.20.04.2) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../09-librbd1_15.2.13-0ubuntu0.20.04.2_amd64.deb ...\r\nUnpacking librbd1 (15.2.13-0ubuntu0.20.04.2) ...\r\nSelecting previously unselected package python3.8-venv.\r\nPreparing to unpack .../10-python3.8-venv_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking python3.8-venv (3.8.10-0ubuntu1~20.04) ...\r\nSelecting previously unselected package python3-venv.\r\nPreparing to unpack .../11-python3-venv_3.8.2-0ubuntu2_amd64.deb ...\r\nUnpacking python3-venv (3.8.2-0ubuntu2) ...\r\nSelecting previously unselected package qemu-block-extra:amd64 2021-09-17T10:02:55.041 INFO:teuthology.task.ansible.out:.\r\nPreparing to unpack .../12-qemu-block-extra_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../13-qemu-system-common_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-system-common (1:4.2-3ubuntu6.17) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../14-qemu-system-x86_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:4.2-3ubuntu6.17) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../15-qemu-utils_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-utils (1:4.2-3ubuntu6.17) ...\r\nSetting up librados2 (15.2.13-0ubuntu0.20.04.2) ...\r\nSetting up libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up git (1:2.25.1-1ubuntu3.2) ...\r\nSetting up librbd1 (15.2.13-0ubuntu0.20.04.2) ...\r\nSetting up python3.8-minimal (3.8.10-0ubuntu1~20.04) ...\r\nSetting up libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up python3.8 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...\r\nSetting up qemu-system-common (1:4.2-3ubuntu6.17) ...\r\nSetting up qemu-system-x86 (1:4.2-3ubuntu6.17) ...\r\nSetting up libpython3.8:amd64 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up python3.8-venv (3.8.10-0ubuntu1~20.04) ...\r\nSetting up qemu-utils (1:4.2-3ubuntu6.17) ...\r\nSetting up python3-venv (3.8.2-0ubuntu2) ...\r\nSetting up libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up python3.8-dev (3.8.10-0ubuntu1~20.04) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for mime-support (3.64ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following additional packages will be installed:", " libpython3.8 libpython3.8-dev libpytho 2021-09-17T10:02:55.043 INFO:teuthology.task.ansible.out:n3.8-minimal libpython3.8-stdlib", " librados2 librbd1 python3.8 python3.8-dev python3.8-minimal python3.8-venv", " qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk", " gitweb git-cvs git-mediawiki git-svn python3.8-doc binfmt-support samba vde2", " debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 python3-venv python3.8-venv qemu-block-extra", " qemu-system-common qemu-system-x86 qemu-utils", "The following packages will be upgraded:", " git libpython3.8 libpython3.8-dev libpython3.8-minimal libpython3.8-stdlib", " python3.8 python3.8-dev python3.8-minimal", "8 upgraded, 8 newly installed, 0 to remove and 167 not upgraded.", "Need to get 29.0 MB of archives.", "After this operation, 62.8 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-dev amd64 3.8.10-0ubuntu1~20.04 [510 kB]", "Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8 amd64 3.8.10-0ubuntu1~20.04 [387 kB]", "Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-dev amd64 3.8.10-0ubuntu1~20.04 [3943 kB]", "Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8 amd64 3.8.10-0ubuntu1~20.04 [1625 kB]", "Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-stdlib amd64 3.8.10-0ubuntu1~20.04 [1675 kB]", "Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-minimal amd64 3.8.10-0ubuntu1~20.04 [1898 kB]", "Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-minimal amd64 3.8.10-0ubuntu1~20.04 [717 kB]", "Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.2 [4554 kB]", "Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.13-0ubuntu0.20.04.2 [3210 kB]", "Get:10 http://archive.ubuntu.com/ubuntu focal 2021-09-17T10:02:55.045 INFO:teuthology.task.ansible.out:-updates/main amd64 librbd1 amd64 15.2.13-0ubuntu0.20.04.2 [1624 kB]", "Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3.8-venv amd64 3.8.10-0ubuntu1~20.04 [5456 B]", "Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-venv amd64 3.8.2-0ubuntu2 [1228 B]", "Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.17 [54.4 kB]", "Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.17 [1055 kB]", "Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.17 [6738 kB]", "Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.17 [973 kB]", "Fetched 29.0 MB in 2s (13.6 MB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138038 files and directories currently installed.)", "Preparing to unpack .../00-python3.8-dev_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking python3.8-dev (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../01-python3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking python3.8 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../02-libpython3.8-dev_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../03-lib 2021-09-17T10:02:55.046 INFO:teuthology.task.ansible.out:python3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking libpython3.8:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../04-libpython3.8-stdlib_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) over (3.8. 2021-09-17T10:02:55.047 INFO:teuthology.task.ansible.out:5-1~20.04.2) ...", "Preparing to unpack .../05-python3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking python3.8-minimal (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../06-libpython3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../07-git_1%3a2.25.1-1ubuntu3.2_amd64.deb ...", "Unpacking git (1:2.25.1-1ubuntu3.2) over (1:2.25.1-1ubuntu3.1) ...", "Selecting previously unselected package librados2.", "Preparing to unpack .../08-librados2_15.2.13-0ubuntu0.20.04.2_amd64.deb ...", "Unpacking librados2 (15.2.13-0ubuntu0.20.04.2) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../09-librbd1_15.2.13-0ubuntu0.20.04.2_amd64.deb ...", "Unpacking librbd1 (15.2.13-0ubuntu0.20.04.2) ...", "Selecting previously unselected package python3.8-venv.", "Preparing to unpack .../10-python3.8-venv_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking python3.8-venv (3.8.10-0ubuntu1~20.04) ...", "Selecting previously unselected package python3-venv.", "Preparing to unpack .../11-python3-venv_3.8.2-0ubuntu2_amd64.deb ...", "Unpacking python3-venv (3.8.2-0ubuntu2) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../12-qemu-block-extra_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../13-qemu-system-common_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-system-common (1:4.2-3ubuntu6.17) ...", "Selecting previously unselected package qemu-system-x86.", 2021-09-17T10:02:55.049 INFO:teuthology.task.ansible.out: "Preparing to unpack .../14-qemu-system-x86_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-system-x86 (1:4.2-3ubuntu6.17) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../15-qemu-utils_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-utils (1:4.2-3ubuntu6.17) ...", "Setting up librados2 (15.2.13-0ubuntu0.20.04.2) ...", "Setting up libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) ...", "Setting up git (1:2.25.1-1ubuntu3.2) ...", "Setting up librbd1 (15.2.13-0ubuntu0.20.04.2) ...", "Setting up python3.8-minimal (3.8.10-0ubuntu1~20.04) ...", "Setting up libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) ...", "Setting up python3.8 (3.8.10-0ubuntu1~20.04) ...", "Setting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...", "Setting up qemu-system-common (1:4.2-3ubuntu6.17) ...", "Setting up qemu-system-x86 (1:4.2-3ubuntu6.17) ...", "Setting up libpython3.8:amd64 (3.8.10-0ubuntu1~20.04) ...", "Setting up python3.8-venv (3.8.10-0ubuntu1~20.04) ...", "Setting up qemu-utils (1:4.2-3ubuntu6.17) ...", "Setting up python3-venv (3.8.2-0ubuntu2) ...", "Setting up libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04) ...", "Setting up python3.8-dev (3.8.10-0ubuntu1~20.04) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for mime-support (3.64ubuntu1) ..."]} 2021-09-17T10:02:55.081 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:55 +0000 (0:00:21.977) 0:02:12.448 ****** 2021-09-17T10:02:55.204 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n libpython3.8 libpython3.8-dev libpython3.8-minimal libpython3.8-stdlib\n librados2 librbd1 python3.8 python3.8-dev python3.8-minimal python3.8-venv\n qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n gitweb git-cvs git-mediawiki git-svn python3.8-doc binfmt-support samba vde2\n debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 python3-venv python3.8-venv qemu-block-extra\n qemu-system-common qemu-system-x86 qemu-utils\nThe following packages will be upgraded:\n git libpython3.8 libpython3.8-dev libpython3.8-minimal libpython3.8-stdlib\n python3.8 python3.8-dev python3.8-minimal\n8 upgraded, 8 newly installed, 0 to remove and 167 not upgraded.\nNeed to get 29.0 MB of archives.\nAfter this operation, 62.8 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-dev amd64 3.8.10-0ubuntu1~20.04 [510 kB]\nGet:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8 amd64 3.8.10-0ubuntu1~20.04 [387 kB]\nGet:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-dev amd64 3.8.10-0ubuntu1~20.04 [3943 kB]\nGet:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8 amd64 3.8.10-0ubuntu1~20.04 [1625 kB]\nGet:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-stdlib amd64 3.8.10-0ubuntu1~20.04 [1675 kB]\nGet:6 http://archive.ubuntu.com/ubuntu focal-u 2021-09-17T10:02:55.206 INFO:teuthology.task.ansible.out:pdates/main amd64 python3.8-minimal amd64 3.8.10-0ubuntu1~20.04 [1898 kB]\nGet:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-minimal amd64 3.8.10-0ubuntu1~20.04 [717 kB]\nGet:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.2 [4554 kB]\nGet:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.13-0ubuntu0.20.04.2 [3210 kB]\nGet:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.13-0ubuntu0.20.04.2 [1624 kB]\nGet:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3.8-venv amd64 3.8.10-0ubuntu1~20.04 [5456 B]\nGet:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-venv amd64 3.8.2-0ubuntu2 [1228 B]\nGet:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.17 [54.4 kB]\nGet:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.17 [1055 kB]\nGet:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.17 [6738 kB]\nGet:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.17 [973 kB]\nFetched 29.0 MB in 2s (12.9 MB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138038 files and directories currently installed.)\r\nPreparing to unpack .../00-python3.8-dev_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking python3.8-de 2021-09-17T10:02:55.208 INFO:teuthology.task.ansible.out:v (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../01-python3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking python3.8 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../02-libpython3.8-dev_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../03-libpython3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking libpython3.8:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../04-libpython3.8-stdlib_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../05-python3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking python3.8-minimal (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../06-libpython3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../07-git_1%3a2.25.1-1ubuntu3.2_amd64.deb ...\r\nUnpacking git (1:2.25.1-1ubuntu3.2) over (1:2.25.1-1ubuntu3.1) ...\r\nSelecting previously unselected package librados2.\r\nPreparing to unpack .../08-librados2_15.2.13-0ubuntu0.20.04.2_amd64.deb ...\r\nUnpacking librados2 (15.2.13-0ubuntu0.20.04.2) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../09-librbd1_15.2.13-0ubuntu0.20.04.2_amd64.deb ...\r\nUnpacking librbd1 (15.2.13-0ubuntu0.20.04.2) ...\r\nSelecting previously unselected package python3.8-venv.\r\nPreparing to unpack .../10-python3.8-venv_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking python3.8-venv (3.8.10-0ubuntu1~20.04) ...\r\nSelecting previously unselected package python3-venv.\r\nPreparing to unpack .../11-python3-venv_3.8.2-0ubuntu2_amd64.deb ...\r\nUnpacking python3-venv (3.8.2-0ubuntu2) ...\r\nSelecting previously unselected package qemu-block-extra:amd64 2021-09-17T10:02:55.209 INFO:teuthology.task.ansible.out:.\r\nPreparing to unpack .../12-qemu-block-extra_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../13-qemu-system-common_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-system-common (1:4.2-3ubuntu6.17) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../14-qemu-system-x86_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:4.2-3ubuntu6.17) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../15-qemu-utils_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-utils (1:4.2-3ubuntu6.17) ...\r\nSetting up librados2 (15.2.13-0ubuntu0.20.04.2) ...\r\nSetting up libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up git (1:2.25.1-1ubuntu3.2) ...\r\nSetting up librbd1 (15.2.13-0ubuntu0.20.04.2) ...\r\nSetting up python3.8-minimal (3.8.10-0ubuntu1~20.04) ...\r\nSetting up libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up python3.8 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...\r\nSetting up qemu-system-common (1:4.2-3ubuntu6.17) ...\r\nSetting up qemu-system-x86 (1:4.2-3ubuntu6.17) ...\r\nSetting up libpython3.8:amd64 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up python3.8-venv (3.8.10-0ubuntu1~20.04) ...\r\nSetting up qemu-utils (1:4.2-3ubuntu6.17) ...\r\nSetting up python3-venv (3.8.2-0ubuntu2) ...\r\nSetting up libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up python3.8-dev (3.8.10-0ubuntu1~20.04) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for mime-support (3.64ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following additional packages will be installed:", " libpython3.8 libpython3.8-dev libpytho 2021-09-17T10:02:55.211 INFO:teuthology.task.ansible.out:n3.8-minimal libpython3.8-stdlib", " librados2 librbd1 python3.8 python3.8-dev python3.8-minimal python3.8-venv", " qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk", " gitweb git-cvs git-mediawiki git-svn python3.8-doc binfmt-support samba vde2", " debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 python3-venv python3.8-venv qemu-block-extra", " qemu-system-common qemu-system-x86 qemu-utils", "The following packages will be upgraded:", " git libpython3.8 libpython3.8-dev libpython3.8-minimal libpython3.8-stdlib", " python3.8 python3.8-dev python3.8-minimal", "8 upgraded, 8 newly installed, 0 to remove and 167 not upgraded.", "Need to get 29.0 MB of archives.", "After this operation, 62.8 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-dev amd64 3.8.10-0ubuntu1~20.04 [510 kB]", "Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8 amd64 3.8.10-0ubuntu1~20.04 [387 kB]", "Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-dev amd64 3.8.10-0ubuntu1~20.04 [3943 kB]", "Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8 amd64 3.8.10-0ubuntu1~20.04 [1625 kB]", "Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-stdlib amd64 3.8.10-0ubuntu1~20.04 [1675 kB]", "Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-minimal amd64 3.8.10-0ubuntu1~20.04 [1898 kB]", "Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-minimal amd64 3.8.10-0ubuntu1~20.04 [717 kB]", "Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.2 [4554 kB]", "Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.13-0ubuntu0.20.04.2 [3210 kB]", "Get:10 http://archive.ubuntu.com/ubuntu focal 2021-09-17T10:02:55.213 INFO:teuthology.task.ansible.out:-updates/main amd64 librbd1 amd64 15.2.13-0ubuntu0.20.04.2 [1624 kB]", "Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3.8-venv amd64 3.8.10-0ubuntu1~20.04 [5456 B]", "Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-venv amd64 3.8.2-0ubuntu2 [1228 B]", "Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.17 [54.4 kB]", "Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.17 [1055 kB]", "Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.17 [6738 kB]", "Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.17 [973 kB]", "Fetched 29.0 MB in 2s (12.9 MB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138038 files and directories currently installed.)", "Preparing to unpack .../00-python3.8-dev_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking python3.8-dev (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../01-python3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking python3.8 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../02-libpython3.8-dev_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../03-lib 2021-09-17T10:02:55.214 INFO:teuthology.task.ansible.out:python3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking libpython3.8:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../04-libpython3.8-stdlib_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../05-python3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking python3.8-minimal (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../06-libpython3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../07-git_1%3a2.25.1-1ubuntu3.2_amd64.deb ...", "Unpacking git (1:2.25.1-1ubuntu3.2) over (1:2.25.1-1ubuntu3.1) ...", "Selecting previously unselected package librados2.", "Preparing to unpack .../08-librados2_15.2.13-0ubuntu0.20.04.2_amd64.deb ...", "Unpacking librados2 (15.2.13-0ubuntu0.20.04.2) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../09-librbd1_15.2.13-0ubuntu0.20.04.2_amd64.deb ...", "Unpacking librbd1 (15.2.13-0ubuntu0.20.04.2) ...", "Selecting previously unselected package python3.8-venv.", "Preparing to unpack .../10-python3.8-venv_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking python3.8-venv (3.8.10-0ubuntu1~20.04) ...", "Selecting previously unselected package python3-venv.", "Preparing to unpack .../11-python3-venv_3.8.2-0ubuntu2_amd64.deb ...", "Unpacking python3-venv (3.8.2-0ubuntu2) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../12-qemu-block-extra_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../13-qemu-system-common_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-system-common (1:4.2-3ubuntu6.17) ...", "Selecting previously unselected package qemu-system-x86.", 2021-09-17T10:02:55.216 INFO:teuthology.task.ansible.out: "Preparing to unpack .../14-qemu-system-x86_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-system-x86 (1:4.2-3ubuntu6.17) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../15-qemu-utils_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-utils (1:4.2-3ubuntu6.17) ...", "Setting up librados2 (15.2.13-0ubuntu0.20.04.2) ...", "Setting up libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) ...", "Setting up git (1:2.25.1-1ubuntu3.2) ...", "Setting up librbd1 (15.2.13-0ubuntu0.20.04.2) ...", "Setting up python3.8-minimal (3.8.10-0ubuntu1~20.04) ...", "Setting up libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) ...", "Setting up python3.8 (3.8.10-0ubuntu1~20.04) ...", "Setting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...", "Setting up qemu-system-common (1:4.2-3ubuntu6.17) ...", "Setting up qemu-system-x86 (1:4.2-3ubuntu6.17) ...", "Setting up libpython3.8:amd64 (3.8.10-0ubuntu1~20.04) ...", "Setting up python3.8-venv (3.8.10-0ubuntu1~20.04) ...", "Setting up qemu-utils (1:4.2-3ubuntu6.17) ...", "Setting up python3-venv (3.8.2-0ubuntu2) ...", "Setting up libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04) ...", "Setting up python3.8-dev (3.8.10-0ubuntu1~20.04) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for mime-support (3.64ubuntu1) ..."]} 2021-09-17T10:02:55.240 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:55 +0000 (0:00:00.159) 0:02:12.607 ****** 2021-09-17T10:02:55.809 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-09-17T10:02:55.814 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:02:55.845 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:55 +0000 (0:00:00.605) 0:02:13.212 ****** 2021-09-17T10:02:56.000 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:02:56.025 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:56 +0000 (0:00:00.179) 0:02:13.392 ****** 2021-09-17T10:02:56.557 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-09-17T10:02:56.559 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:02:56.583 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:56 +0000 (0:00:00.557) 0:02:13.950 ****** 2021-09-17T10:02:56.760 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:02:56.762 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-09-17T10:02:56.785 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:56 +0000 (0:00:00.202) 0:02:14.153 ****** 2021-09-17T10:02:56.818 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:56 +0000 (0:00:00.031) 0:02:14.185 ****** 2021-09-17T10:02:56.874 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-09-17T10:02:56.876 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:56.905 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:56 +0000 (0:00:00.085) 0:02:14.271 ****** 2021-09-17T10:02:57.024 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-09-17T10:02:57.027 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-09-17T10:02:57.050 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:57 +0000 (0:00:00.146) 0:02:14.418 ****** 2021-09-17T10:02:57.093 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-09-17T10:02:57.120 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:57 +0000 (0:00:00.069) 0:02:14.487 ****** 2021-09-17T10:02:57.138 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:57.164 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:57 +0000 (0:00:00.044) 0:02:14.531 ****** 2021-09-17T10:02:57.342 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-09-17T10:02:57.344 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-09-17T10:02:57.370 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:57 +0000 (0:00:00.203) 0:02:14.735 ****** 2021-09-17T10:02:57.529 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-09-17T10:02:57.554 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:57 +0000 (0:00:00.186) 0:02:14.921 ****** 2021-09-17T10:02:57.773 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-09-17T10:02:57.775 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-09-17T10:02:57.799 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:57 +0000 (0:00:00.244) 0:02:15.166 ****** 2021-09-17T10:02:57.821 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:57 +0000 (0:00:00.021) 0:02:15.188 ****** 2021-09-17T10:02:57.977 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi018.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T10:02:58.002 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:57 +0000 (0:00:00.181) 0:02:15.369 ****** 2021-09-17T10:02:58.061 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-09-17T10:02:58.063 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-09-17T10:02:58.090 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.088) 0:02:15.457 ****** 2021-09-17T10:02:58.285 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-09-17T10:02:58.287 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T10:02:58.310 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.220) 0:02:15.678 ****** 2021-09-17T10:02:58.319 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-09-17T10:02:58.321 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Fri 2021-09-17 10:00:04 UTC", "ActiveEnterTimestampMonotonic": "16455804", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target system.slice basic.target systemd-tmpfiles-setup.service nss-lookup.target -.mount network.target sysinit.target systemd-journald.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-09-17 10:00:03 UTC", "AssertTimestampMonotonic": "15651656", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-09-17 10:00:03 UTC", "ConditionTimestampMonotonic": "15651656", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup" 2021-09-17T10:02:58.322 INFO:teuthology.task.ansible.out:: "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "Documentation": "https://httpd.apache.org/docs/2.4/", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "886", "ExecMainStartTimestamp": "Fri 2021-09-17 10:00:04 UTC", "ExecMainStartTimestampMonotonic": "16455764", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709 2021-09-17T10:02:58.324 INFO:teuthology.task.ansible.out:551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-09-17 10:00:03 UTC", "InactiveExitTimestampMonotonic": "15652688", "InvocationID": "1d0f53896554434983d95f774a8c063f", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127802", "LimitNPROCSoft": "127802", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127802", "LimitSIGPENDINGSoft": "127802", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "886", "MemoryAccounting": "yes", "MemoryCurrent": "10215424", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolic 2021-09-17T10:02:58.326 INFO:teuthology.task.ansible.out:y": "n/a", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-09-17 10:00:04 UTC", "StateChangeTimestampMonotonic": "16455804", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "38340", "TimeoutAbortUSec": 2021-09-17T10:02:58.327 INFO:teuthology.task.ansible.out:"1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-17T10:02:58.355 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.040) 0:02:15.718 ****** 2021-09-17T10:02:58.425 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-17T10:02:58.427 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:58.460 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.108) 0:02:15.827 ****** 2021-09-17T10:02:58.532 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-17T10:02:58.534 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:58.559 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.098) 0:02:15.926 ****** 2021-09-17T10:02:58.630 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-09-17T10:02:58.631 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Fri 2021-09-17 10:00:12 UTC", "ActiveEnterTimestampMonotonic": "17962878", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice network.target nss-lookup.target systemd-journald.socket remote-fs.target basic.target -.mount sysinit.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-09-17 10:00:10 UTC", "AssertTimestampMonotonic": "16748647", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-09-17 10:00:10 UTC", "ConditionTimestampMonotonic": "16748647", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup" 2021-09-17T10:02:58.633 INFO:teuthology.task.ansible.out:: "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "Documentation": "https://httpd.apache.org/docs/2.4/", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "903", "ExecMainStartTimestamp": "Fri 2021-09-17 10:00:12 UTC", "ExecMainStartTimestampMonotonic": "17962839", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709 2021-09-17T10:02:58.634 INFO:teuthology.task.ansible.out:551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-09-17 10:00:10 UTC", "InactiveExitTimestampMonotonic": "16749685", "InvocationID": "6bd1f9e042cb4a0587de9dc5dea51221", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127802", "LimitNPROCSoft": "127802", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127802", "LimitSIGPENDINGSoft": "127802", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "903", "MemoryAccounting": "yes", "MemoryCurrent": "10018816", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolic 2021-09-17T10:02:58.636 INFO:teuthology.task.ansible.out:y": "n/a", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-09-17 10:00:12 UTC", "StateChangeTimestampMonotonic": "17962878", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "38340", "TimeoutAbortUSec": 2021-09-17T10:02:58.637 INFO:teuthology.task.ansible.out:"1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:58.657 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.098) 0:02:16.024 ****** 2021-09-17T10:02:58.681 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.023) 0:02:16.048 ****** 2021-09-17T10:02:58.730 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-17T10:02:58.732 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:58.757 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.075) 0:02:16.124 ****** 2021-09-17T10:02:58.764 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-17T10:02:58.765 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:58.791 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.034) 0:02:16.158 ****** 2021-09-17T10:02:58.830 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:58.854 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.063) 0:02:16.222 ****** 2021-09-17T10:02:58.867 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-17T10:02:58.869 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:58.893 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.038) 0:02:16.261 ****** 2021-09-17T10:02:58.931 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-09-17T10:02:58.933 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:58.956 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.062) 0:02:16.324 ****** 2021-09-17T10:02:58.975 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-09-17T10:02:58.976 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.000 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:58 +0000 (0:00:00.043) 0:02:16.367 ****** 2021-09-17T10:02:59.027 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-17T10:02:59.029 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.055 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.055) 0:02:16.423 ****** 2021-09-17T10:02:59.083 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-17T10:02:59.085 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.110 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.054) 0:02:16.477 ****** 2021-09-17T10:02:59.132 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-17T10:02:59.133 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.165 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.054) 0:02:16.532 ****** 2021-09-17T10:02:59.186 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-17T10:02:59.188 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.215 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.049) 0:02:16.582 ****** 2021-09-17T10:02:59.244 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T10:02:59.246 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.272 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.056) 0:02:16.639 ****** 2021-09-17T10:02:59.290 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.317 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.045) 0:02:16.684 ****** 2021-09-17T10:02:59.335 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-17T10:02:59.337 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.363 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.046) 0:02:16.730 ****** 2021-09-17T10:02:59.391 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-17T10:02:59.392 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.416 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.053) 0:02:16.784 ****** 2021-09-17T10:02:59.434 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-17T10:02:59.436 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.461 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.044) 0:02:16.829 ****** 2021-09-17T10:02:59.490 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-17T10:02:59.492 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.516 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.054) 0:02:16.884 ****** 2021-09-17T10:02:59.544 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-17T10:02:59.546 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.570 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.053) 0:02:16.937 ****** 2021-09-17T10:02:59.587 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T10:02:59.589 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.616 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.046) 0:02:16.984 ****** 2021-09-17T10:02:59.645 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-17T10:02:59.646 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.680 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.062) 0:02:17.046 ****** 2021-09-17T10:02:59.696 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-17T10:02:59.698 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.724 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.043) 0:02:17.090 ****** 2021-09-17T10:02:59.751 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-17T10:02:59.752 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.779 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.056) 0:02:17.146 ****** 2021-09-17T10:02:59.798 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-17T10:02:59.800 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.827 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.048) 0:02:17.195 ****** 2021-09-17T10:02:59.846 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-17T10:02:59.848 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.875 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.047) 0:02:17.242 ****** 2021-09-17T10:02:59.894 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-17T10:02:59.896 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.920 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.045) 0:02:17.288 ****** 2021-09-17T10:02:59.950 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:02:59.977 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:02:59 +0000 (0:00:00.056) 0:02:17.344 ****** 2021-09-17T10:02:59.996 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-17T10:02:59.998 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.022 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.045) 0:02:17.389 ****** 2021-09-17T10:03:00.050 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-17T10:03:00.052 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.078 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.054) 0:02:17.443 ****** 2021-09-17T10:03:00.094 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-17T10:03:00.096 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.121 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.043) 0:02:17.487 ****** 2021-09-17T10:03:00.148 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-17T10:03:00.150 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.174 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.053) 0:02:17.541 ****** 2021-09-17T10:03:00.201 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-17T10:03:00.203 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.228 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.054) 0:02:17.595 ****** 2021-09-17T10:03:00.245 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-17T10:03:00.247 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.275 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.046) 0:02:17.642 ****** 2021-09-17T10:03:00.305 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.332 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.057) 0:02:17.700 ****** 2021-09-17T10:03:00.349 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-17T10:03:00.351 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.378 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.045) 0:02:17.745 ****** 2021-09-17T10:03:00.395 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-17T10:03:00.397 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.423 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.044) 0:02:17.790 ****** 2021-09-17T10:03:00.439 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-17T10:03:00.441 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.466 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.043) 0:02:17.833 ****** 2021-09-17T10:03:00.484 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-17T10:03:00.485 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.511 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.045) 0:02:17.879 ****** 2021-09-17T10:03:00.528 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-17T10:03:00.530 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.556 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.044) 0:02:17.923 ****** 2021-09-17T10:03:00.585 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-17T10:03:00.587 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.611 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.055) 0:02:17.978 ****** 2021-09-17T10:03:00.628 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-17T10:03:00.629 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.653 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.042) 0:02:18.020 ****** 2021-09-17T10:03:00.681 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-17T10:03:00.683 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.707 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.053) 0:02:18.074 ****** 2021-09-17T10:03:00.723 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-17T10:03:00.725 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.749 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.042) 0:02:18.116 ****** 2021-09-17T10:03:00.777 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-17T10:03:00.779 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.802 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.053) 0:02:18.169 ****** 2021-09-17T10:03:00.819 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-17T10:03:00.821 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.849 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.046) 0:02:18.216 ****** 2021-09-17T10:03:00.880 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-17T10:03:00.882 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.907 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.058) 0:02:18.274 ****** 2021-09-17T10:03:00.925 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.951 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.044) 0:02:18.319 ****** 2021-09-17T10:03:00.969 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:00.995 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:00 +0000 (0:00:00.043) 0:02:18.362 ****** 2021-09-17T10:03:01.066 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-17T10:03:01.068 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:01.090 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:01 +0000 (0:00:00.095) 0:02:18.458 ****** 2021-09-17T10:03:01.129 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-09-17T10:03:01.131 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-09-17T10:03:01.156 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:01 +0000 (0:00:00.065) 0:02:18.523 ****** 2021-09-17T10:03:01.163 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-17T10:03:01.165 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:01.190 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:01 +0000 (0:00:00.034) 0:02:18.557 ****** 2021-09-17T10:03:01.271 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-09-17T10:03:01.272 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:01.295 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:01 +0000 (0:00:00.104) 0:02:18.662 ****** 2021-09-17T10:03:01.482 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-09-17T10:03:01.484 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-09-17T10:03:01.508 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:01 +0000 (0:00:00.212) 0:02:18.875 ****** 2021-09-17T10:03:01.535 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-17T10:03:01.537 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-09-17T10:03:01.562 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:01 +0000 (0:00:00.053) 0:02:18.928 ****** 2021-09-17T10:03:01.641 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:01.666 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:01 +0000 (0:00:00.105) 0:02:19.033 ****** 2021-09-17T10:03:01.895 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-17T10:03:01.896 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-09-17T10:03:01.924 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:01 +0000 (0:00:00.257) 0:02:19.291 ****** 2021-09-17T10:03:01.994 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-17T10:03:01.995 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:02.019 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:02 +0000 (0:00:00.095) 0:02:19.386 ****** 2021-09-17T10:03:02.069 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-09-17T10:03:02.100 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:02 +0000 (0:00:00.080) 0:02:19.467 ****** 2021-09-17T10:03:02.418 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-09-17T10:03:02.445 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:02 +0000 (0:00:00.344) 0:02:19.812 ****** 2021-09-17T10:03:02.494 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-09-17T10:03:02.495 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-09-17T10:03:02.520 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:02 +0000 (0:00:00.074) 0:02:19.887 ****** 2021-09-17T10:03:02.841 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-09-17T10:03:02.865 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:02 +0000 (0:00:00.345) 0:02:20.232 ****** 2021-09-17T10:03:02.904 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-09-17T10:03:02.906 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-09-17T10:03:02.930 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:02 +0000 (0:00:00.065) 0:02:20.298 ****** 2021-09-17T10:03:03.116 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-09-17T10:03:03.118 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T10:03:03.140 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.209) 0:02:20.508 ****** 2021-09-17T10:03:03.220 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-09-17T10:03:03.222 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.245 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.104) 0:02:20.613 ****** 2021-09-17T10:03:03.252 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-09-17T10:03:03.253 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-09-17T10:03:03.279 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.033) 0:02:20.646 ****** 2021-09-17T10:03:03.317 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-09-17T10:03:03.319 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.347 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.067) 0:02:20.714 ****** 2021-09-17T10:03:03.428 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-09-17T10:03:03.429 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.454 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.107) 0:02:20.821 ****** 2021-09-17T10:03:03.472 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-09-17T10:03:03.473 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T10:03:03.500 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.046) 0:02:20.867 ****** 2021-09-17T10:03:03.517 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-09-17T10:03:03.519 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.544 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.044) 0:02:20.911 ****** 2021-09-17T10:03:03.574 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.601 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.057) 0:02:20.969 ****** 2021-09-17T10:03:03.620 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-09-17T10:03:03.622 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.649 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.047) 0:02:21.016 ****** 2021-09-17T10:03:03.678 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-09-17T10:03:03.680 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.708 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.059) 0:02:21.075 ****** 2021-09-17T10:03:03.726 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T10:03:03.728 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-09-17T10:03:03.753 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.044) 0:02:21.120 ****** 2021-09-17T10:03:03.780 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-09-17T10:03:03.782 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.806 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.053) 0:02:21.173 ****** 2021-09-17T10:03:03.845 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi018.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.846 INFO:teuthology.task.ansible.out: 2021-09-17T10:03:03.858 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.869 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.881 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.882 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** ok: [smithi018.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-09-17T10:03:03.906 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.099) 0:02:21.273 ****** 2021-09-17T10:03:03.913 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.913 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.914 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.914 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.926 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.928 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.937 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:03.966 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:03 +0000 (0:00:00.060) 0:02:21.333 ****** 2021-09-17T10:03:03.983 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-09-17T10:03:03.985 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.011 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:04 +0000 (0:00:00.044) 0:02:21.378 ****** 2021-09-17T10:03:04.101 INFO:teuthology.task.ansible.out: 2021-09-17T10:03:04.103 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-09-17T10:03:04.131 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:04 +0000 (0:00:00.120) 0:02:21.498 ****** 2021-09-17T10:03:04.210 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-09-17T10:03:04.212 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.234 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.244 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.246 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003677", "end": "2021-09-17 10:03:04.217013", "rc": 0, "start": "2021-09-17 10:03:04.213336", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_4e1f59de82924f116bd6b91664f90ef632c60a3f/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_4e1f59de82924f116bd6b91664f90ef632c60a3f/virtualenv/bin/teuthology-lock"]} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi050.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.274 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:04 +0000 (0:00:00.142) 0:02:21.641 ****** 2021-09-17T10:03:04.283 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-09-17T10:03:04.284 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.284 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.286 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.296 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.298 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.309 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.338 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:04 +0000 (0:00:00.064) 0:02:21.705 ****** 2021-09-17T10:03:04.355 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-09-17T10:03:04.357 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.383 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:04 +0000 (0:00:00.045) 0:02:21.751 ****** 2021-09-17T10:03:04.452 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-09-17T10:03:04.455 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.478 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:04 +0000 (0:00:00.094) 0:02:21.845 ****** 2021-09-17T10:03:04.549 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.582 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:04 +0000 (0:00:00.104) 0:02:21.950 ****** 2021-09-17T10:03:04.591 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-09-17T10:03:04.593 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003377", "end": "2021-09-17 10:03:04.557715", "rc": 0, "start": "2021-09-17 10:03:04.554338", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_4e1f59de82924f116bd6b91664f90ef632c60a3f/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_4e1f59de82924f116bd6b91664f90ef632c60a3f/virtualenv/bin/teuthology-lock"]} 2021-09-17T10:03:04.626 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:04 +0000 (0:00:00.042) 0:02:21.992 ****** 2021-09-17T10:03:04.707 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-09-17T10:03:04.709 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.735 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:04 +0000 (0:00:00.109) 0:02:22.102 ****** 2021-09-17T10:03:04.815 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.840 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:04 +0000 (0:00:00.104) 0:02:22.207 ****** 2021-09-17T10:03:04.920 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-09-17T10:03:04.922 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:04.945 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:04 +0000 (0:00:00.105) 0:02:22.312 ****** 2021-09-17T10:03:05.311 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-09-17T10:03:05.314 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:03:05.336 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:05 +0000 (0:00:00.391) 0:02:22.704 ****** 2021-09-17T10:03:05.408 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:05.435 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:05 +0000 (0:00:00.098) 0:02:22.802 ****** 2021-09-17T10:03:05.514 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-17T10:03:05.515 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item={'block_used': 13871133, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226223975, 'size_available': 926613401600, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150395, 'block_size': 4096, 'inode_available': 60904581}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226223975, "block_size": 4096, "block_total": 240095108, "block_used": 13871133, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904581, "inode_total": 61054976, "inode_used": 150395, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926613401600, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}} 2021-09-17T10:03:05.539 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:05 +0000 (0:00:00.104) 0:02:22.907 ****** 2021-09-17T10:03:05.682 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-09-17T10:03:05.684 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:03:05.709 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:05 +0000 (0:00:00.169) 0:02:23.076 ****** 2021-09-17T10:03:05.728 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-09-17T10:03:05.730 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009588", "end": "2021-09-17 10:03:05.346678", "rc": 0, "start": "2021-09-17 10:03:05.337090", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:05.757 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:05 +0000 (0:00:00.047) 0:02:23.124 ****** 2021-09-17T10:03:05.775 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-09-17T10:03:05.777 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:05.803 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:05 +0000 (0:00:00.046) 0:02:23.170 ****** 2021-09-17T10:03:05.833 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-09-17T10:03:05.859 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'block_used': 13871133, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226223975, 'size_available': 926613401600, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150395, 'block_size': 4096, 'inode_available': 60904581}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226223975, "block_size": 4096, "block_total": 240095108, "block_used": 13871133, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904581, "inode_total": 61054976, "inode_used": 150395, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926613401600, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:05.860 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:05 +0000 (0:00:00.056) 0:02:23.227 ****** 2021-09-17T10:03:05.888 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-17T10:03:05.890 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item={'block_used': 13871133, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226223975, 'size_available': 926613401600, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150395, 'block_size': 4096, 'inode_available': 60904581}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226223975, "block_size": 4096, "block_total": 240095108, "block_used": 13871133, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904581, "inode_total": 61054976, "inode_used": 150395, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926613401600, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}} 2021-09-17T10:03:05.918 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:05 +0000 (0:00:00.056) 0:02:23.283 ****** 2021-09-17T10:03:05.935 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-09-17T10:03:05.963 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:05 +0000 (0:00:00.047) 0:02:23.331 ****** 2021-09-17T10:03:06.056 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:06.109 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-09-17T10:03:06.111 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009311", "end": "2021-09-17 10:03:07.389667", "rc": 0, "start": "2021-09-17 10:03:07.380356", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:06.134 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:06 +0000 (0:00:00.171) 0:02:23.502 ****** 2021-09-17T10:03:06.214 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-09-17T10:03:06.216 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'block_used': 13871133, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226223975, 'size_available': 926613401600, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150395, 'block_size': 4096, 'inode_available': 60904581}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226223975, "block_size": 4096, "block_total": 240095108, "block_used": 13871133, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904581, "inode_total": 61054976, "inode_used": 150395, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926613401600, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:06.239 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:06 +0000 (0:00:00.104) 0:02:23.607 ****** 2021-09-17T10:03:06.309 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-09-17T10:03:06.334 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:06 +0000 (0:00:00.095) 0:02:23.702 ****** 2021-09-17T10:03:06.425 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-09-17T10:03:06.427 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:07.891 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2qKvbyjBmV0bE4ybScK824pwWo15NBwQN'], 'uuids': ['1d41f190-587c-4491-9d75-afc05681bda1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.710111", "end": "2021-09-17 10:03:07.500884", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2qKvbyjBmV0bE4ybScK824pwWo15NBwQN"], "labels": [], "masters": [], "uuids": ["1d41f190-587c-4491-9d75-afc05681bda1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:05.790773", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T10:03:07.903 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:07.914 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:08.341 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg1MlUb2juyAQWLJXcV9m3emSlxB8tk5IU'], 'uuids': ['08269a7c-3f5d-4705-ae6c-8065ee3e6d85']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.787656", "end": "2021-09-17 10:03:09.612077", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg1MlUb2juyAQWLJXcV9m3emSlxB8tk5IU"], "labels": [], "masters": [], "uuids": ["08269a7c-3f5d-4705-ae6c-8065ee3e6d85"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:07.824421", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T10:03:08.352 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:08.364 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:09.145 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2f0B945yiIslwEY24COAERUMsihIgdOfL'], 'uuids': ['0f63575e-1d17-4418-8121-1aa05855cc5e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.100506", "end": "2021-09-17 10:03:08.752975", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2f0B945yiIslwEY24COAERUMsihIgdOfL"], "labels": [], "masters": [], "uuids": ["0f63575e-1d17-4418-8121-1aa05855cc5e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:07.652469", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T10:03:09.158 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQDQ', 'wwn-0x5000c500871a8c35'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQDQ-part1', 'wwn-0x5000c500871a8c35-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a8c35', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HQDQ', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQDQ", "wwn-0x5000c500871a8c35"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQDQ-part1", "wwn-0x5000c500871a8c35-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQDQ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a8c35"}}, 2021-09-17T10:03:09.159 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2021-09-17T10:03:09.615 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg3x9Nyxq9moDOr3kfOrahbidn2DiVKifH'], 'uuids': ['9e64ff9a-2135-470f-b977-06893b4262c6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.122595", "end": "2021-09-17 10:03:10.882960", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg3x9Nyxq9moDOr3kfOrahbidn2DiVKifH"], "labels": [], "masters": [], "uuids": ["9e64ff9a-2135-470f-b977-06893b4262c6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:09.760365", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T10:03:09.627 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HNFD', 'wwn-0x5000c500871ad2da'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HNFD-part1', 'wwn-0x5000c500871ad2da-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871ad2da', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HNFD', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HNFD", "wwn-0x5000c500871ad2da"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HNFD-part1", "wwn-0x5000c500871ad2da-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HNFD", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871ad2da"}}, 2021-09-17T10:03:09.629 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2021-09-17T10:03:10.426 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-2jv1PA-r9t7-TtNR-hpPK-C0iH-xMna-Rx6Vva', 'nvme-INTEL_SSDPED1D480GA_PHMB75120091480DGN', 'nvme-nvme.8086-50484d42373531323030393134383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.140170", "end": "2021-09-17 10:03:10.031398", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-2jv1PA-r9t7-TtNR-hpPK-C0iH-xMna-Rx6Vva", "nvme-INTEL_SSDPED1D480GA_PHMB75120091480DGN", "nvme-nvme.8086-50484d42373531323030393134383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:08.891228", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nGPT data structures destroyed! You 2021-09-17T10:03:10.428 INFO:teuthology.task.ansible.out:may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:10.439 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:11.012 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-Kg3YXE-AUDX-7RCD-7vaG-55zI-blro-A8w2mk', 'nvme-INTEL_SSDPED1D480GA_PHMB7513001F480DGN', 'nvme-nvme.8086-50484d42373531333030314634383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.258803", "end": "2021-09-17 10:03:12.279770", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-Kg3YXE-AUDX-7RCD-7vaG-55zI-blro-A8w2mk", "nvme-INTEL_SSDPED1D480GA_PHMB7513001F480DGN", "nvme-nvme.8086-50484d42373531333030314634383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:11.020967", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nGPT data structures destroyed! You 2021-09-17T10:03:11.014 INFO:teuthology.task.ansible.out:may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T10:03:11.025 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:11.026 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:11.766 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x233IO1YO9IPYsBKUBxRBIIDMrdV2zL59k'], 'uuids': ['4414cb1c-03b3-4c3c-a32d-e554f494d266']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.198127", "end": "2021-09-17 10:03:11.373896", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x233IO1YO9IPYsBKUBxRBIIDMrdV2zL59k"], "labels": [], "masters": [], "uuids": ["4414cb1c-03b3-4c3c-a32d-e554f494d266"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:10.175769", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T10:03:11.778 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:11.779 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:12.322 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg25JomrCbSgrf8tQVUL6q2tTKhQBc20gj'], 'uuids': ['cb597abb-fe10-473e-a709-fe383f04cd3b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.166689", "end": "2021-09-17 10:03:13.593565", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg25JomrCbSgrf8tQVUL6q2tTKhQBc20gj"], "labels": [], "masters": [], "uuids": ["cb597abb-fe10-473e-a709-fe383f04cd3b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:12.426876", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T10:03:12.335 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:12.336 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:13.055 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2YyqT35ovjDITI9V9GWzRIjiHo7Gq78CP'], 'uuids': ['41e0450b-a3e1-4509-b007-ef6ce931e605']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.134786", "end": "2021-09-17 10:03:12.658598", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2YyqT35ovjDITI9V9GWzRIjiHo7Gq78CP"], "labels": [], "masters": [], "uuids": ["41e0450b-a3e1-4509-b007-ef6ce931e605"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:11.523812", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T10:03:13.066 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:13.588 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDgCCWmBjT0fzWfQK936A1HfJUv9Eryw5HB'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.111298", "end": "2021-09-17 10:03:14.852390", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDgCCWmBjT0fzWfQK936A1HfJUv9Eryw5HB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:13.741092", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T10:03:13.599 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:14.329 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2D5HHUtRlfUBkczwx3CTu5zoSLxvVEJm7'], 'uuids': ['e74f1fe5-b744-4f0b-8dca-5635a05a67ff']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.137394", "end": "2021-09-17 10:03:13.941061", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2D5HHUtRlfUBkczwx3CTu5zoSLxvVEJm7"], "labels": [], "masters": [], "uuids": ["e74f1fe5-b744-4f0b-8dca-5635a05a67ff"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:12.803667", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T10:03:14.373 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:14 +0000 (0:00:08.038) 0:02:31.741 ****** 2021-09-17T10:03:14.464 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-17T10:03:14.466 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:14.707 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2qKvbyjBmV0bE4ybScK824pwWo15NBwQN'], 'uuids': ['1d41f190-587c-4491-9d75-afc05681bda1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.118029", "end": "2021-09-17 10:03:14.317489", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2qKvbyjBmV0bE4ybScK824pwWo15NBwQN"], "labels": [], "masters": [], "uuids": ["1d41f190-587c-4491-9d75-afc05681bda1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:14.199460", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.113057 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.113057 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:14.720 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:14.722 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:14.848 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-09-17T10:03:14.849 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg8alQfGOCebSLJ5S9k5Xa1oSep26kBRdo'], 'uuids': ['e7ad3921-22ab-4948-a34a-b4522e71a55b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.122052", "end": "2021-09-17 10:03:16.115781", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg8alQfGOCebSLJ5S9k5Xa1oSep26kBRdo"], "labels": [], "masters": [], "uuids": ["e7ad3921-22ab-4948-a34a-b4522e71a55b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:14.993729", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-17T10:03:14.883 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:14 +0000 (0:00:00.510) 0:02:32.251 ****** 2021-09-17T10:03:14.963 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-17T10:03:14.965 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2f0B945yiIslwEY24COAERUMsihIgdOfL'], 'uuids': ['0f63575e-1d17-4418-8121-1aa05855cc5e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.112477", "end": "2021-09-17 10:03:14.571036", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2f0B945yiIslwEY24COAERUMsihIgdOfL"], "labels": [], "masters": [], "uuids": ["0f63575e-1d17-4418-8121-1aa05855cc5e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:14.458559", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.107677 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.107677 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:14.965 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQDQ', 'wwn-0x5000c500871a8c35'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], ' 2021-09-17T10:03:14.967 INFO:teuthology.task.ansible.out:labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQDQ-part1', 'wwn-0x5000c500871a8c35-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a8c35', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HQDQ', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQDQ", "wwn-0x5000c500871a8c35"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HQDQ-part1", "wwn-0x5000c500871a8c35-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HQDQ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a8c35"}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:14.988 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:15.156 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-2jv1PA-r9t7-TtNR-hpPK-C0iH-xMna-Rx6Vva', 'nvme-INTEL_SSDPED1D480GA_PHMB75120091480DGN', 'nvme-nvme.8086-50484d42373531323030393134383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.056468", "end": "2021-09-17 10:03:14.765387", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-2jv1PA-r9t7-TtNR-hpPK-C0iH-xMna-Rx6Vva", "nvme-INTEL_SSDPED1D480GA_PHMB75120091480DGN", "nvme-nvme.8086-50484d42373531323030393134383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:14.708919", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0516813 s, 2.2 GB/s", "stderr_lines": [" 2021-09-17T10:03:15.158 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0516813 s, 2.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:15.170 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:15.254 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg1MlUb2juyAQWLJXcV9m3emSlxB8tk5IU'], 'uuids': ['08269a7c-3f5d-4705-ae6c-8065ee3e6d85']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.117904", "end": "2021-09-17 10:03:16.518915", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg1MlUb2juyAQWLJXcV9m3emSlxB8tk5IU"], "labels": [], "masters": [], "uuids": ["08269a7c-3f5d-4705-ae6c-8065ee3e6d85"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:16.401011", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.113005 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.113005 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:15.255 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:15.267 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:15.403 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x233IO1YO9IPYsBKUBxRBIIDMrdV2zL59k'], 'uuids': ['4414cb1c-03b3-4c3c-a32d-e554f494d266']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.111750", "end": "2021-09-17 10:03:15.015449", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x233IO1YO9IPYsBKUBxRBIIDMrdV2zL59k"], "labels": [], "masters": [], "uuids": ["4414cb1c-03b3-4c3c-a32d-e554f494d266"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:14.903699", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.107135 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.107135 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:15.414 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:15.426 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:15.510 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg3x9Nyxq9moDOr3kfOrahbidn2DiVKifH'], 'uuids': ['9e64ff9a-2135-470f-b977-06893b4262c6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.114218", "end": "2021-09-17 10:03:16.778505", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg3x9Nyxq9moDOr3kfOrahbidn2DiVKifH"], "labels": [], "masters": [], "uuids": ["9e64ff9a-2135-470f-b977-06893b4262c6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:16.664287", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.109341 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.109341 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:15.522 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HNFD', 'wwn-0x5000c500871ad2da'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HNFD-part1', 'wwn-0x5000c500871ad2da-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871ad2da', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HNFD', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HNFD", "wwn-0x5000c500871ad2da"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HNFD-part1", "wwn-0x5000c500871ad2da-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HNFD", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871ad2da"}}, 2021-09-17T10:03:15.526 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2021-09-17T10:03:15.670 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2YyqT35ovjDITI9V9GWzRIjiHo7Gq78CP'], 'uuids': ['41e0450b-a3e1-4509-b007-ef6ce931e605']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.118550", "end": "2021-09-17 10:03:15.277854", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2YyqT35ovjDITI9V9GWzRIjiHo7Gq78CP"], "labels": [], "masters": [], "uuids": ["41e0450b-a3e1-4509-b007-ef6ce931e605"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:15.159304", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.113593 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.113593 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:15.672 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:15.692 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-Kg3YXE-AUDX-7RCD-7vaG-55zI-blro-A8w2mk', 'nvme-INTEL_SSDPED1D480GA_PHMB7513001F480DGN', 'nvme-nvme.8086-50484d42373531333030314634383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.052203", "end": "2021-09-17 10:03:16.967722", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-Kg3YXE-AUDX-7RCD-7vaG-55zI-blro-A8w2mk", "nvme-INTEL_SSDPED1D480GA_PHMB7513001F480DGN", "nvme-nvme.8086-50484d42373531333030314634383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:16.915519", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0474243 s, 2.4 GB/s", "stderr_lines": [" 2021-09-17T10:03:15.694 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0474243 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:15.704 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:15.715 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:15.914 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2D5HHUtRlfUBkczwx3CTu5zoSLxvVEJm7'], 'uuids': ['e74f1fe5-b744-4f0b-8dca-5635a05a67ff']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.118658", "end": "2021-09-17 10:03:15.530715", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Dp8Ki7Z2wSwy1OveDPeix5kGom4gu6x2D5HHUtRlfUBkczwx3CTu5zoSLxvVEJm7"], "labels": [], "masters": [], "uuids": ["e74f1fe5-b744-4f0b-8dca-5635a05a67ff"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:15.412057", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.113798 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.113798 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:15.960 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:15 +0000 (0:00:01.076) 0:02:33.327 ****** 2021-09-17T10:03:15.968 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg25JomrCbSgrf8tQVUL6q2tTKhQBc20gj'], 'uuids': ['cb597abb-fe10-473e-a709-fe383f04cd3b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.114378", "end": "2021-09-17 10:03:17.227168", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg25JomrCbSgrf8tQVUL6q2tTKhQBc20gj"], "labels": [], "masters": [], "uuids": ["cb597abb-fe10-473e-a709-fe383f04cd3b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:17.112790", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.109794 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.109794 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:15.970 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:15.983 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:16.219 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDgCCWmBjT0fzWfQK936A1HfJUv9Eryw5HB'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.118182", "end": "2021-09-17 10:03:17.493213", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDgCCWmBjT0fzWfQK936A1HfJUv9Eryw5HB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:17.375031", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.113386 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.113386 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:16.230 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:16.295 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-09-17T10:03:16.297 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.148938", "end": "2021-09-17 10:03:15.859763", "rc": 0, "start": "2021-09-17 10:03:15.710825", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:16.321 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:16 +0000 (0:00:00.361) 0:02:33.689 ****** 2021-09-17T10:03:16.464 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-17T10:03:16.466 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg8alQfGOCebSLJ5S9k5Xa1oSep26kBRdo'], 'uuids': ['e7ad3921-22ab-4948-a34a-b4522e71a55b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.113823", "end": "2021-09-17 10:03:17.738869", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-oeStO3DSSebY2ozpkksis8BlPWNQMGDg8alQfGOCebSLJ5S9k5Xa1oSep26kBRdo"], "labels": [], "masters": [], "uuids": ["e7ad3921-22ab-4948-a34a-b4522e71a55b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-17 10:03:17.625046", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.108965 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.108965 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:16.514 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:16 +0000 (0:00:00.192) 0:02:33.881 ****** 2021-09-17T10:03:16.543 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-09-17T10:03:16.545 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.075753", "end": "2021-09-17 10:03:16.156279", "rc": 0, "start": "2021-09-17 10:03:16.080526", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:16.574 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:16 +0000 (0:00:00.060) 0:02:33.942 ****** 2021-09-17T10:03:16.644 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-09-17T10:03:16.711 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:16 +0000 (0:00:00.136) 0:02:34.078 ****** 2021-09-17T10:03:16.739 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:16.766 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:16 +0000 (0:00:00.055) 0:02:34.134 ****** 2021-09-17T10:03:16.796 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-09-17T10:03:16.799 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:16.822 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:16 +0000 (0:00:00.055) 0:02:34.189 ****** 2021-09-17T10:03:16.828 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-09-17T10:03:16.830 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.132384", "end": "2021-09-17 10:03:18.059798", "rc": 0, "start": "2021-09-17 10:03:17.927414", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:16.854 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:16 +0000 (0:00:00.031) 0:02:34.221 ****** 2021-09-17T10:03:16.861 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-17T10:03:16.863 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:16.892 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:16 +0000 (0:00:00.038) 0:02:34.259 ****** 2021-09-17T10:03:16.920 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-09-17T10:03:16.921 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:16.944 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:16 +0000 (0:00:00.052) 0:02:34.311 ****** 2021-09-17T10:03:17.067 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-09-17T10:03:17.069 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.071910", "end": "2021-09-17 10:03:18.340566", "rc": 0, "start": "2021-09-17 10:03:18.268656", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:17.091 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.147) 0:02:34.459 ****** 2021-09-17T10:03:17.098 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-09-17T10:03:17.100 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T10:03:17.125 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.033) 0:02:34.492 ****** 2021-09-17T10:03:17.131 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-09-17T10:03:17.162 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.037) 0:02:34.529 ****** 2021-09-17T10:03:17.168 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-17T10:03:17.170 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'block_used': 13871133, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226223975, 'size_available': 926613401600, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150395, 'block_size': 4096, 'inode_available': 60904581}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226223975, "block_size": 4096, "block_total": 240095108, "block_used": 13871133, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904581, "inode_total": 61054976, "inode_used": 150395, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926613401600, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.196 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.034) 0:02:34.563 ****** 2021-09-17T10:03:17.204 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-09-17T10:03:17.206 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.234 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.037) 0:02:34.601 ****** 2021-09-17T10:03:17.242 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-09-17T10:03:17.243 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.275 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.039) 0:02:34.641 ****** 2021-09-17T10:03:17.280 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.311 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.037) 0:02:34.678 ****** 2021-09-17T10:03:17.319 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T10:03:17.320 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.344 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.033) 0:02:34.712 ****** 2021-09-17T10:03:17.351 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-17T10:03:17.353 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.380 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.035) 0:02:34.747 ****** 2021-09-17T10:03:17.389 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T10:03:17.390 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.417 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.037) 0:02:34.785 ****** 2021-09-17T10:03:17.426 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-09-17T10:03:17.428 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.452 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.034) 0:02:34.819 ****** 2021-09-17T10:03:17.459 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-09-17T10:03:17.461 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.487 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.034) 0:02:34.854 ****** 2021-09-17T10:03:17.514 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-09-17T10:03:17.516 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.542 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.054) 0:02:34.909 ****** 2021-09-17T10:03:17.570 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-09-17T10:03:17.573 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.595 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.053) 0:02:34.962 ****** 2021-09-17T10:03:17.604 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-09-17T10:03:17.605 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-17T10:03:17.640 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.044) 0:02:35.007 ****** 2021-09-17T10:03:17.677 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-17T10:03:17.679 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'block_used': 13871133, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226223975, 'size_available': 926613401600, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150395, 'block_size': 4096, 'inode_available': 60904581}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226223975, "block_size": 4096, "block_total": 240095108, "block_used": 13871133, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904581, "inode_total": 61054976, "inode_used": 150395, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926613401600, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.702 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.062) 0:02:35.069 ****** 2021-09-17T10:03:17.733 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-09-17T10:03:17.735 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.759 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.057) 0:02:35.126 ****** 2021-09-17T10:03:17.789 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.812 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.053) 0:02:35.180 ****** 2021-09-17T10:03:17.815 INFO:teuthology.task.ansible.out: 2021-09-17T10:03:17.854 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.882 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.068) 0:02:35.249 ****** 2021-09-17T10:03:17.921 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:17.949 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:17 +0000 (0:00:00.067) 0:02:35.316 ****** 2021-09-17T10:03:17.987 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-09-17T10:03:17.989 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:18.012 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:18 +0000 (0:00:00.062) 0:02:35.379 ****** 2021-09-17T10:03:18.040 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-09-17T10:03:18.041 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:18.069 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:18 +0000 (0:00:00.057) 0:02:35.437 ****** 2021-09-17T10:03:18.389 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-09-17T10:03:18.415 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:18 +0000 (0:00:00.345) 0:02:35.782 ****** 2021-09-17T10:03:18.625 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-09-17T10:03:18.659 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:18 +0000 (0:00:00.244) 0:02:36.026 ****** 2021-09-17T10:03:19.091 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-17T10:03:19.093 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T10:03:19.145 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T10:03:19.474 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T10:03:19.559 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T10:03:19.908 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T10:03:19.962 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T10:03:20.305 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T10:03:20.369 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T10:03:20.695 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T10:03:20.725 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:20 +0000 (0:00:02.066) 0:02:38.093 ****** 2021-09-17T10:03:20.766 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-17T10:03:20.795 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:20 +0000 (0:00:00.069) 0:02:38.163 ****** 2021-09-17T10:03:20.877 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-17T10:03:20.879 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-09-17T10:03:20.904 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:20 +0000 (0:00:00.108) 0:02:38.271 ****** 2021-09-17T10:03:20.942 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-09-17T10:03:20.969 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:20 +0000 (0:00:00.065) 0:02:38.337 ****** 2021-09-17T10:03:21.065 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T10:03:21.138 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T10:03:21.193 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T10:03:21.258 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T10:03:21.310 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T10:03:21.387 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T10:03:21.430 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-17T10:03:21.442 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:21.472 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:21 +0000 (0:00:00.502) 0:02:38.840 ****** 2021-09-17T10:03:21.503 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:21.539 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:21 +0000 (0:00:00.064) 0:02:38.905 ****** 2021-09-17T10:03:21.549 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-17T10:03:21.552 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} skipping: [smithi050.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:21.585 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:21 +0000 (0:00:00.047) 0:02:38.952 ****** 2021-09-17T10:03:21.595 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-09-17T10:03:21.597 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:21.633 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:21 +0000 (0:00:00.047) 0:02:39.000 ****** 2021-09-17T10:03:21.643 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-09-17T10:03:21.645 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:21.692 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:21 +0000 (0:00:00.058) 0:02:39.058 ****** 2021-09-17T10:03:21.722 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:21.747 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:21 +0000 (0:00:00.054) 0:02:39.113 ****** 2021-09-17T10:03:21.785 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-09-17T10:03:21.787 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T10:03:21.811 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:21 +0000 (0:00:00.065) 0:02:39.179 ****** 2021-09-17T10:03:21.862 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-09-17T10:03:21.864 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-09-17T10:03:21.895 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:21 +0000 (0:00:00.083) 0:02:39.262 ****** 2021-09-17T10:03:21.902 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-09-17T10:03:21.904 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T10:03:21.929 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:21 +0000 (0:00:00.033) 0:02:39.296 ****** 2021-09-17T10:03:21.962 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-09-17T10:03:21.964 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-09-17T10:03:21.988 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:21 +0000 (0:00:00.059) 0:02:39.355 ****** 2021-09-17T10:03:22.313 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-09-17T10:03:22.315 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true} changed: [smithi050.front.sepia.ceph.com] => {"changed": true} 2021-09-17T10:03:22.340 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:22 +0000 (0:00:00.352) 0:02:39.708 ****** 2021-09-17T10:03:22.363 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:22 +0000 (0:00:00.022) 0:02:39.730 ****** 2021-09-17T10:03:22.797 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-09-17T10:03:22.820 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:22 +0000 (0:00:00.456) 0:02:40.187 ****** 2021-09-17T10:03:22.827 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-09-17T10:03:22.853 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:22 +0000 (0:00:00.032) 0:02:40.220 ****** 2021-09-17T10:03:22.860 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-09-17T10:03:22.862 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:22.889 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:22 +0000 (0:00:00.035) 0:02:40.256 ****** 2021-09-17T10:03:22.896 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:22.926 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:22 +0000 (0:00:00.036) 0:02:40.292 ****** 2021-09-17T10:03:22.933 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-09-17T10:03:22.935 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:22.960 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:22 +0000 (0:00:00.034) 0:02:40.327 ****** 2021-09-17T10:03:22.968 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:22.996 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:22 +0000 (0:00:00.035) 0:02:40.362 ****** 2021-09-17T10:03:23.003 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-09-17T10:03:23.005 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:23.030 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:23 +0000 (0:00:00.034) 0:02:40.397 ****** 2021-09-17T10:03:23.037 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:23.067 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:23 +0000 (0:00:00.037) 0:02:40.434 ****** 2021-09-17T10:03:23.076 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:23.102 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:23 +0000 (0:00:00.034) 0:02:40.469 ****** 2021-09-17T10:03:23.110 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:23.139 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:23 +0000 (0:00:00.035) 0:02:40.504 ****** 2021-09-17T10:03:23.281 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-09-17T10:03:23.282 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-09-17T10:03:23.307 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:23 +0000 (0:00:00.169) 0:02:40.674 ****** 2021-09-17T10:03:23.316 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-09-17T10:03:23.344 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:23 +0000 (0:00:00.036) 0:02:40.711 ****** 2021-09-17T10:03:23.466 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-09-17T10:03:23.468 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T10:03:23.493 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:23 +0000 (0:00:00.148) 0:02:40.860 ****** 2021-09-17T10:03:23.563 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T10:03:23.588 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:23 +0000 (0:00:00.095) 0:02:40.956 ****** 2021-09-17T10:03:23.857 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-09-17T10:03:23.859 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-09-17T10:03:23.888 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:23 +0000 (0:00:00.299) 0:02:41.255 ****** 2021-09-17T10:03:23.970 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-09-17T10:03:23.996 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:23 +0000 (0:00:00.108) 0:02:41.363 ****** 2021-09-17T10:03:24.246 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-09-17T10:03:24.273 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:24 +0000 (0:00:00.277) 0:02:41.641 ****** 2021-09-17T10:03:24.357 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-09-17T10:03:24.385 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:24 +0000 (0:00:00.111) 0:02:41.752 ****** 2021-09-17T10:03:24.971 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:03:25.008 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:25 +0000 (0:00:00.622) 0:02:42.375 ****** 2021-09-17T10:03:25.084 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:03:25.113 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:25 +0000 (0:00:00.105) 0:02:42.481 ****** 2021-09-17T10:03:25.290 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-09-17T10:03:25.292 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.119230", "end": "2021-09-17 10:03:25.275512", "rc": 0, "start": "2021-09-17 10:03:25.156282", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.30.0/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.30.0/Amazon/S3.pm"]} 2021-09-17T10:03:25.318 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:25 +0000 (0:00:00.204) 0:02:42.685 ****** 2021-09-17T10:03:25.347 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:25.371 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:25 +0000 (0:00:00.052) 0:02:42.738 ****** 2021-09-17T10:03:25.399 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-09-17T10:03:25.401 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.127699", "end": "2021-09-17 10:03:26.670715", "rc": 0, "start": "2021-09-17 10:03:26.543016", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.30.0/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.30.0/Amazon/S3.pm"]} TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:25.428 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:25 +0000 (0:00:00.056) 0:02:42.795 ****** 2021-09-17T10:03:25.451 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:25 +0000 (0:00:00.023) 0:02:42.819 ****** 2021-09-17T10:03:25.492 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-09-17T10:03:25.494 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:25.517 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:25 +0000 (0:00:00.066) 0:02:42.885 ****** 2021-09-17T10:03:25.547 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-09-17T10:03:25.549 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:25.571 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:25 +0000 (0:00:00.053) 0:02:42.939 ****** 2021-09-17T10:03:26.116 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-09-17T10:03:26.118 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:03:26.153 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:26 +0000 (0:00:00.581) 0:02:43.520 ****** 2021-09-17T10:03:26.182 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:26.209 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:26 +0000 (0:00:00.055) 0:02:43.576 ****** 2021-09-17T10:03:26.272 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-09-17T10:03:26.273 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:03:26.311 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:26 +0000 (0:00:00.102) 0:02:43.678 ****** 2021-09-17T10:03:26.352 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-09-17T10:03:26.353 INFO:teuthology.task.ansible.out: skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:26.384 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:26 +0000 (0:00:00.070) 0:02:43.749 ****** 2021-09-17T10:03:26.673 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-09-17T10:03:26.675 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-09-17T10:03:26.699 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:26 +0000 (0:00:00.317) 0:02:44.067 ****** 2021-09-17T10:03:26.727 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-09-17T10:03:26.729 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:26.751 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:26 +0000 (0:00:00.051) 0:02:44.118 ****** 2021-09-17T10:03:26.832 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-09-17T10:03:26.835 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-09-17T10:03:26.872 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:26 +0000 (0:00:00.120) 0:02:44.239 ****** 2021-09-17T10:03:26.901 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-09-17T10:03:26.903 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:26.930 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:26 +0000 (0:00:00.058) 0:02:44.298 ****** 2021-09-17T10:03:27.023 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-09-17T10:03:27.027 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-09-17 10:00:04 UTC", "ActiveEnterTimestampMonotonic": "15796864", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount basic.target systemd-tmpfiles-setup.service sysinit.target systemd-journald.socket network.target system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-09-17 10:00:03 UTC", "AssertTimestampMonotonic": "15599456", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-09-17 10:00:03 UTC", "ConditionTimestampMonotonic": "15599454", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": 2021-09-17T10:03:27.028 INFO:teuthology.task.ansible.out: "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "824", "ExecMainStartTimestamp": "Fri 2021-09-17 10:00:04 UTC", "ExecMainStartTimestampMonotonic": "15796850", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-09-17 10:00:03 UTC", "InactiveExitTimestampMonotonic": "15600648", "InvocationID": "1660688f341f464e8749edd8f385587c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "Lim 2021-09-17T10:03:27.030 INFO:teuthology.task.ansible.out:itCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127802", "LimitNPROCSoft": "127802", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127802", "LimitSIGPENDINGSoft": "127802", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "824", "MemoryAccounting": "yes", "MemoryCurrent": "3608576", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "Requi 2021-09-17T10:03:27.031 INFO:teuthology.task.ansible.out:resMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-09-17 10:00:04 UTC", "StateChangeTimestampMonotonic": "15796864", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "38340", "TimeoutAbortUSec": "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-17T10:03:27.064 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:27 +0000 (0:00:00.133) 0:02:44.431 ****** 2021-09-17T10:03:27.103 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T10:03:27.105 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-09-17T10:03:27.127 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:27 +0000 (0:00:00.063) 0:02:44.495 ****** 2021-09-17T10:03:27.169 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-09-17T10:03:27.171 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-09-17 10:00:11 UTC", "ActiveEnterTimestampMonotonic": "16884626", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target systemd-tmpfiles-setup.service system.slice network.target basic.target -.mount systemd-journald.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-09-17 10:00:10 UTC", "AssertTimestampMonotonic": "16684412", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-09-17 10:00:10 UTC", "ConditionTimestampMonotonic": "16684412", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": 2021-09-17T10:03:27.173 INFO:teuthology.task.ansible.out: "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "824", "ExecMainStartTimestamp": "Fri 2021-09-17 10:00:11 UTC", "ExecMainStartTimestampMonotonic": "16884613", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-09-17 10:00:10 UTC", "InactiveExitTimestampMonotonic": "16685442", "InvocationID": "40ef386c1e634cd6be0bcfa9a03c5cc5", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "Lim 2021-09-17T10:03:27.175 INFO:teuthology.task.ansible.out:itCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127802", "LimitNPROCSoft": "127802", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127802", "LimitSIGPENDINGSoft": "127802", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "824", "MemoryAccounting": "yes", "MemoryCurrent": "3575808", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "Requi 2021-09-17T10:03:27.176 INFO:teuthology.task.ansible.out:resMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-09-17 10:00:11 UTC", "StateChangeTimestampMonotonic": "16884626", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "38340", "TimeoutAbortUSec": "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} TASK [testnode : set_fact] ***************************************************** ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-09-17T10:03:27.200 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:27 +0000 (0:00:00.072) 0:02:44.568 ****** 2021-09-17T10:03:27.223 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:27 +0000 (0:00:00.022) 0:02:44.591 ****** 2021-09-17T10:03:27.242 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-09-17T10:03:27.244 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:27.268 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:27 +0000 (0:00:00.044) 0:02:44.635 ****** 2021-09-17T10:03:27.276 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T10:03:27.278 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-09-17T10:03:27.305 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:27 +0000 (0:00:00.036) 0:02:44.672 ****** 2021-09-17T10:03:27.343 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T10:03:27.346 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-09-17T10:03:27.369 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:27 +0000 (0:00:00.064) 0:02:44.737 ****** 2021-09-17T10:03:27.397 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-09-17T10:03:27.399 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:27.422 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:27 +0000 (0:00:00.052) 0:02:44.789 ****** 2021-09-17T10:03:28.212 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-09-17T10:03:28.214 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:03:28.238 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:28 +0000 (0:00:00.816) 0:02:45.605 ****** 2021-09-17T10:03:28.266 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-09-17T10:03:28.269 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:28.291 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:28 +0000 (0:00:00.052) 0:02:45.658 ****** 2021-09-17T10:03:28.362 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-09-17T10:03:28.364 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:03:28.391 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:28 +0000 (0:00:00.099) 0:02:45.758 ****** 2021-09-17T10:03:28.430 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-09-17T10:03:28.432 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:28.456 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:28 +0000 (0:00:00.064) 0:02:45.823 ****** 2021-09-17T10:03:28.475 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-09-17T10:03:28.477 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-09-17T10:03:28.502 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:28 +0000 (0:00:00.045) 0:02:45.869 ****** 2021-09-17T10:03:28.656 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-09-17T10:03:28.683 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:28 +0000 (0:00:00.181) 0:02:46.051 ****** 2021-09-17T10:03:28.954 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-09-17T10:03:28.957 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-09-17T10:03:28.980 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:28 +0000 (0:00:00.297) 0:02:46.348 ****** 2021-09-17T10:03:29.116 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-09-17T10:03:29.144 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:29 +0000 (0:00:00.162) 0:02:46.511 ****** 2021-09-17T10:03:31.222 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)\nRequirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)", "Requirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)"], "version": null, "virtualenv": null} 2021-09-17T10:03:31.246 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:31 +0000 (0:00:02.101) 0:02:48.613 ****** 2021-09-17T10:03:31.264 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)\nRequirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)", "Requirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)"], "version": null, "virtualenv": null} 2021-09-17T10:03:31.291 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:31 +0000 (0:00:00.045) 0:02:48.658 ****** 2021-09-17T10:03:31.790 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-09-17T10:03:31.792 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false} 2021-09-17T10:03:31.820 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:31 +0000 (0:00:00.528) 0:02:49.187 ****** 2021-09-17T10:03:31.839 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false} 2021-09-17T10:03:31.865 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:31 +0000 (0:00:00.045) 0:02:49.232 ****** 2021-09-17T10:03:31.874 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-09-17T10:03:31.876 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-09-17T10:03:31.901 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:31 +0000 (0:00:00.036) 0:02:49.269 ****** 2021-09-17T10:03:31.909 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-09-17T10:03:31.941 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:31 +0000 (0:00:00.039) 0:02:49.309 ****** 2021-09-17T10:03:32.090 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-09-17T10:03:32.092 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1631872556.1225045, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1631872556.1185045, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 105181, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1631872556.1185045, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "1956422440", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-09-17T10:03:32.122 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:32 +0000 (0:00:00.179) 0:02:49.488 ****** 2021-09-17T10:03:32.140 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1631872557.0549245, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1631872557.0509245, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 105181, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1631872557.0509245, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "1956422440", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-09-17T10:03:32.168 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:32 +0000 (0:00:00.046) 0:02:49.534 ****** 2021-09-17T10:03:32.538 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-09-17T10:03:32.540 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "msg": ""} ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-09-17T10:03:32.564 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:32 +0000 (0:00:00.397) 0:02:49.932 ****** 2021-09-17T10:03:32.590 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:32 +0000 (0:00:00.025) 0:02:49.957 ****** 2021-09-17T10:03:32.609 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-09-17T10:03:32.611 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-09-17T10:03:32.635 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:32 +0000 (0:00:00.044) 0:02:50.002 ****** 2021-09-17T10:03:32.644 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-09-17T10:03:32.670 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:32 +0000 (0:00:00.035) 0:02:50.037 ****** 2021-09-17T10:03:32.677 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:32.706 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:32 +0000 (0:00:00.036) 0:02:50.073 ****** 2021-09-17T10:03:32.713 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:32.739 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:32 +0000 (0:00:00.033) 0:02:50.107 ****** 2021-09-17T10:03:32.893 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-09-17T10:03:32.895 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-09-17T10:03:32.921 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:32 +0000 (0:00:00.180) 0:02:50.287 ****** 2021-09-17T10:03:32.938 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-09-17T10:03:32.964 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:32 +0000 (0:00:00.044) 0:02:50.332 ****** 2021-09-17T10:03:33.087 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-09-17T10:03:33.089 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-09-17T10:03:33.113 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.148) 0:02:50.480 ****** 2021-09-17T10:03:33.142 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-09-17T10:03:33.144 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi050.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-09-17T10:03:33.170 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.057) 0:02:50.537 ****** 2021-09-17T10:03:33.193 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.022) 0:02:50.560 ****** 2021-09-17T10:03:33.209 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-09-17T10:03:33.211 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=docker0) => {"ansible_loop_var": "item", "changed": false, "item": "docker0", "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.222 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-09-17T10:03:33.224 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi018.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.247 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.054) 0:02:50.615 ****** 2021-09-17T10:03:33.254 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.282 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.034) 0:02:50.649 ****** 2021-09-17T10:03:33.290 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=docker0) => {"ansible_loop_var": "item", "changed": false, "item": "docker0", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.301 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.303 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.313 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.314 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.315 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-09-17T10:03:33.315 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'docker0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "docker0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.341 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.059) 0:02:50.708 ****** 2021-09-17T10:03:33.349 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.352 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi018.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.353 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.381 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.040) 0:02:50.749 ****** 2021-09-17T10:03:33.389 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'docker0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "docker0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.401 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.403 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi050.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.403 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.405 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T10:03:33.407 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.434 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.052) 0:02:50.801 ****** 2021-09-17T10:03:33.455 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.021) 0:02:50.823 ****** 2021-09-17T10:03:33.462 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-09-17T10:03:33.463 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.488 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.032) 0:02:50.856 ****** 2021-09-17T10:03:33.495 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-17T10:03:33.497 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.523 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.034) 0:02:50.890 ****** 2021-09-17T10:03:33.530 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-09-17T10:03:33.531 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.562 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.038) 0:02:50.929 ****** 2021-09-17T10:03:33.570 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-09-17T10:03:33.572 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.596 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.033) 0:02:50.963 ****** 2021-09-17T10:03:33.603 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-09-17T10:03:33.605 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.636 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.039) 0:02:51.003 ****** 2021-09-17T10:03:33.643 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-09-17T10:03:33.646 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.672 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.035) 0:02:51.039 ****** 2021-09-17T10:03:33.678 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-09-17T10:03:33.680 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.707 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.035) 0:02:51.074 ****** 2021-09-17T10:03:33.714 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-09-17T10:03:33.717 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.741 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.034) 0:02:51.108 ****** 2021-09-17T10:03:33.748 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-17T10:03:33.749 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.778 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.036) 0:02:51.145 ****** 2021-09-17T10:03:33.785 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-09-17T10:03:33.786 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.813 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.035) 0:02:51.181 ****** 2021-09-17T10:03:33.820 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-09-17T10:03:33.822 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.848 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.034) 0:02:51.215 ****** 2021-09-17T10:03:33.855 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-17T10:03:33.857 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.882 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.034) 0:02:51.250 ****** 2021-09-17T10:03:33.890 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-17T10:03:33.891 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.916 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.033) 0:02:51.284 ****** 2021-09-17T10:03:33.923 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-09-17T10:03:33.925 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:33.953 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:33 +0000 (0:00:00.036) 0:02:51.320 ****** 2021-09-17T10:03:33.981 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-17T10:03:33.983 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:34.006 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:34 +0000 (0:00:00.053) 0:02:51.374 ****** 2021-09-17T10:03:34.117 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi018.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010444", "end": "2021-09-17 10:03:34.105411", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-09-17 10:03:34.094967", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:34.208 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010832", "end": "2021-09-17 10:03:34.196399", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-09-17 10:03:34.185567", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:34.281 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006060", "end": "2021-09-17 10:03:34.273325", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-09-17 10:03:34.267265", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:34.375 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005998", "end": "2021-09-17 10:03:34.364519", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-09-17 10:03:34.358521", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:34.439 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005625", "end": "2021-09-17 10:03:34.434071", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-09-17 10:03:34.428446", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:34.534 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005418", "end": "2021-09-17 10:03:34.520602", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-09-17 10:03:34.515184", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:34.646 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.029427", "end": "2021-09-17 10:03:34.619979", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-09-17 10:03:34.590552", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:34.679 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:34 +0000 (0:00:00.672) 0:02:52.046 ****** 2021-09-17T10:03:34.708 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-09-17T10:03:34.732 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:34 +0000 (0:00:00.052) 0:02:52.099 ****** 2021-09-17T10:03:34.739 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-09-17T10:03:34.741 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.032608", "end": "2021-09-17 10:03:34.708954", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-09-17 10:03:34.676346", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:34.767 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:34 +0000 (0:00:00.035) 0:02:52.134 ****** 2021-09-17T10:03:34.774 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-09-17T10:03:34.775 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:34.799 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-09-17T10:03:34.833 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:34 +0000 (0:00:00.065) 0:02:52.200 ****** 2021-09-17T10:03:34.860 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-09-17T10:03:34.861 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:34.931 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-09-17T10:03:34.952 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:34 +0000 (0:00:00.119) 0:02:52.320 ****** 2021-09-17T10:03:34.975 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:34 +0000 (0:00:00.022) 0:02:52.342 ****** 2021-09-17T10:03:36.123 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi018.front.sepia.ceph.com] 2021-09-17T10:03:36.134 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] 2021-09-17T10:03:36.197 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:36 +0000 (0:00:01.222) 0:02:53.564 ****** 2021-09-17T10:03:36.220 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:36 +0000 (0:00:00.023) 0:02:53.587 ****** 2021-09-17T10:03:36.238 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-17T10:03:36.240 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-09-17T10:03:36.263 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:36 +0000 (0:00:00.042) 0:02:53.630 ****** 2021-09-17T10:03:36.270 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-09-17T10:03:36.296 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:36 +0000 (0:00:00.033) 0:02:53.664 ****** 2021-09-17T10:03:36.303 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-17T10:03:36.305 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:36.329 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:36 +0000 (0:00:00.032) 0:02:53.696 ****** 2021-09-17T10:03:36.336 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:36.371 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:36 +0000 (0:00:00.042) 0:02:53.739 ****** 2021-09-17T10:03:36.378 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-09-17T10:03:36.380 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-09-17T10:03:36.407 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:36 +0000 (0:00:00.035) 0:02:53.774 ****** 2021-09-17T10:03:36.425 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-09-17T10:03:36.452 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:36 +0000 (0:00:00.045) 0:02:53.819 ****** 2021-09-17T10:03:37.209 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-09-17T10:03:37.211 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:03:37.234 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:37 +0000 (0:00:00.782) 0:02:54.602 ****** 2021-09-17T10:03:37.241 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:03:37.266 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:37 +0000 (0:00:00.031) 0:02:54.633 ****** 2021-09-17T10:03:37.272 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-17T10:03:37.273 INFO:teuthology.task.ansible.out:skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:37.300 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:37 +0000 (0:00:00.034) 0:02:54.668 ****** 2021-09-17T10:03:37.308 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:37.335 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:37 +0000 (0:00:00.034) 0:02:54.702 ****** 2021-09-17T10:03:37.341 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-17T10:03:37.343 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-09-17T10:03:37.368 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:37 +0000 (0:00:00.032) 0:02:54.735 ****** 2021-09-17T10:03:37.375 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-09-17T10:03:37.402 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:37 +0000 (0:00:00.033) 0:02:54.769 ****** 2021-09-17T10:03:37.575 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-09-17T10:03:37.576 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T10:03:37.607 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T10:03:37.732 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T10:03:37.760 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:37 +0000 (0:00:00.358) 0:02:55.128 ****** 2021-09-17T10:03:37.778 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-09-17T10:03:37.805 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:37 +0000 (0:00:00.045) 0:02:55.173 ****** 2021-09-17T10:03:38.187 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-09-17T10:03:38.189 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "size": 2334, "state": "file", "uid": 0} 2021-09-17T10:03:38.241 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "size": 2334, "state": "file", "uid": 0} 2021-09-17T10:03:38.576 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "size": 2334, "state": "file", "uid": 0} 2021-09-17T10:03:38.603 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:38 +0000 (0:00:00.797) 0:02:55.970 ****** 2021-09-17T10:03:38.632 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "size": 2334, "state": "file", "uid": 0} 2021-09-17T10:03:38.663 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:38 +0000 (0:00:00.060) 0:02:56.030 ****** 2021-09-17T10:03:39.391 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-09-17T10:03:39.393 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"cache_update_time": 1631872587, "cache_updated": false, "changed": false} 2021-09-17T10:03:39.415 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:39 +0000 (0:00:00.751) 0:02:56.782 ****** 2021-09-17T10:03:39.454 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"cache_update_time": 1631872651, "cache_updated": false, "changed": false} 2021-09-17T10:03:39.481 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:39 +0000 (0:00:00.065) 0:02:56.848 ****** 2021-09-17T10:03:43.486 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": " Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-xtu1f7ui\n", "stderr_lines": [" Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-xtu1f7ui"], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-xtu1f7ui\nRequirement already satisfied, skipping upgrade: docopt in /usr/local/lib/python3.8/dist-packages (from registries-conf-ctl==1.0) (0.6.2)\nRequirement already satisfied, skipping upgrade: toml in /usr/local/lib/python3.8/dist-packages (from registries-conf-ctl==1.0) (0.10.2)\nBuilding wheels for collected packages: registries-conf-ctl\n Building wheel for registries-conf-ctl (setup.py): started\n Building wheel for registries-conf-ctl (setup.py): finished with status 'done'\n Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=8180 sha256=206ec4a6695804f1ba33978e5b697b1697135fa41bae7945ef749646b4e27c52\n Stored in directory: /tmp/pip-ephem-wheel-cache-6capwogj/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42\nSuccessfully built registries-conf-ctl\nInstalling collected packages: registries-conf-ctl\n Attempting uninstall: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebasti 2021-09-17T10:03:43.488 INFO:teuthology.task.ansible.out:an-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-xtu1f7ui", "Requirement already satisfied, skipping upgrade: docopt in /usr/local/lib/python3.8/dist-packages (from registries-conf-ctl==1.0) (0.6.2)", "Requirement already satisfied, skipping upgrade: toml in /usr/local/lib/python3.8/dist-packages (from registries-conf-ctl==1.0) (0.10.2)", "Building wheels for collected packages: registries-conf-ctl", " Building wheel for registries-conf-ctl (setup.py): started", " Building wheel for registries-conf-ctl (setup.py): finished with status 'done'", " Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=8180 sha256=206ec4a6695804f1ba33978e5b697b1697135fa41bae7945ef749646b4e27c52", " Stored in directory: /tmp/pip-ephem-wheel-cache-6capwogj/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42", "Successfully built registries-conf-ctl", "Installing collected packages: registries-conf-ctl", " Attempting uninstall: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-09-17T10:03:43.512 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:43 +0000 (0:00:04.030) 0:03:00.879 ****** 2021-09-17T10:03:43.646 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": " Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-px4om4hv\n", "stderr_lines": [" Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-px4om4hv"], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-px4om4hv\nRequirement already satisfied, skipping upgrade: docopt in /usr/local/lib/python3.8/dist-packages (from registries-conf-ctl==1.0) (0.6.2)\nRequirement already satisfied, skipping upgrade: toml in /usr/local/lib/python3.8/dist-packages (from registries-conf-ctl==1.0) (0.10.2)\nBuilding wheels for collected packages: registries-conf-ctl\n Building wheel for registries-conf-ctl (setup.py): started\n Building wheel for registries-conf-ctl (setup.py): finished with status 'done'\n Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=8180 sha256=206ec4a6695804f1ba33978e5b697b1697135fa41bae7945ef749646b4e27c52\n Stored in directory: /tmp/pip-ephem-wheel-cache-fx4ijjpy/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42\nSuccessfully built registries-conf-ctl\nInstalling collected packages: registries-conf-ctl\n Attempting uninstall: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/reg 2021-09-17T10:03:43.648 INFO:teuthology.task.ansible.out:istries-conf-ctl to ./pip-req-build-px4om4hv", "Requirement already satisfied, skipping upgrade: docopt in /usr/local/lib/python3.8/dist-packages (from registries-conf-ctl==1.0) (0.6.2)", "Requirement already satisfied, skipping upgrade: toml in /usr/local/lib/python3.8/dist-packages (from registries-conf-ctl==1.0) (0.10.2)", "Building wheels for collected packages: registries-conf-ctl", " Building wheel for registries-conf-ctl (setup.py): started", " Building wheel for registries-conf-ctl (setup.py): finished with status 'done'", " Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=8180 sha256=206ec4a6695804f1ba33978e5b697b1697135fa41bae7945ef749646b4e27c52", " Stored in directory: /tmp/pip-ephem-wheel-cache-fx4ijjpy/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42", "Successfully built registries-conf-ctl", "Installing collected packages: registries-conf-ctl", " Attempting uninstall: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-09-17T10:03:43.672 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:43 +0000 (0:00:00.159) 0:03:01.039 ****** 2021-09-17T10:03:43.700 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-09-17T10:03:43.702 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1631872546.8389246, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "checksum": "76c6651ed619dc361f25958c4bc22bbed7de3326", "ctime": 1628780204.6025991, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 95289, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "application/json", "mode": "0644", "mtime": 1628780204.6025991, "nlink": 1, "path": "/etc/docker/daemon.json", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 73, "uid": 0, "version": "2929411125", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-09-17T10:03:43.729 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:43 +0000 (0:00:00.057) 0:03:01.096 ****** 2021-09-17T10:03:43.767 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ skipping: [smithi018.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:43.792 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:43 +0000 (0:00:00.063) 0:03:01.159 ****** 2021-09-17T10:03:43.862 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-09-17T10:03:43.864 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1631872545.0345042, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "checksum": "76c6651ed619dc361f25958c4bc22bbed7de3326", "ctime": 1628780204.6025991, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 95289, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "application/json", "mode": "0644", "mtime": 1628780204.6025991, "nlink": 1, "path": "/etc/docker/daemon.json", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 73, "uid": 0, "version": "2929411125", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-09-17T10:03:43.889 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:43 +0000 (0:00:00.096) 0:03:01.256 ****** 2021-09-17T10:03:43.926 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-09-17T10:03:43.928 INFO:teuthology.task.ansible.out:skipping: [smithi050.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-17T10:03:43.951 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:43 +0000 (0:00:00.062) 0:03:01.319 ****** 2021-09-17T10:03:44.072 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-09-17T10:03:44.074 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.088665", "end": "2021-09-17 10:03:44.062410", "rc": 0, "start": "2021-09-17 10:03:43.973745", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:44.099 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:44 +0000 (0:00:00.147) 0:03:01.466 ****** 2021-09-17T10:03:44.223 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.073449", "end": "2021-09-17 10:03:44.206787", "rc": 0, "start": "2021-09-17 10:03:44.133338", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-17T10:03:44.248 INFO:teuthology.task.ansible.out:Friday 17 September 2021 10:03:44 +0000 (0:00:00.149) 0:03:01.615 ****** 2021-09-17T10:03:45.758 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-09-17T10:03:45.759 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-09-17 10:00:08 UTC", "ActiveEnterTimestampMonotonic": "20543978", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target system.slice network-online.target containerd.service basic.target firewalld.service docker.socket systemd-journald.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-09-17 10:00:04 UTC", "AssertTimestampMonotonic": "15801797", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-09-17 10:00:04 UTC", "ConditionTimestampMonotonic": "15801796", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/ 2021-09-17T10:03:45.761 INFO:teuthology.task.ansible.out:docker.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "yes", "DelegateControllers": "cpu cpuacct cpuset io blkio memory devices pids bpf-firewall bpf-devices", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "854", "ExecMainStartTimestamp": "Fri 2021-09-17 10:00:04 UTC", "ExecMainStartTimestampMonotonic": "15804039", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "docker.service", "IgnoreOnIso 2021-09-17T10:03:45.762 INFO:teuthology.task.ansible.out:late": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-09-17 10:00:04 UTC", "InactiveExitTimestampMonotonic": "15804270", "InvocationID": "0a7b4b2640304993bbe4600b135cd640", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127802", "LimitSIGPENDINGSoft": "127802", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "854", "MemoryAccounting": "yes", "MemoryCurrent": "137252864", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMPolicy": "continue", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace" 2021-09-17T10:03:45.764 INFO:teuthology.task.ansible.out:, "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target docker.socket", "Restart": "always", "RestartKillSignal": "15", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-09-17 10:00:08 UTC", "StateChangeTimestampMonotonic": "20543978", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "17", "TasksMax": "infinity", "TimeoutAbortUSec": "infinity", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset 2021-09-17T10:03:45.766 INFO:teuthology.task.ansible.out:": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "Wants": "containerd.service network-online.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-17T10:03:45.809 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-09-17 10:00:15 UTC", "ActiveEnterTimestampMonotonic": "21472266", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target basic.target containerd.service systemd-journald.socket docker.socket firewalld.service system.slice network-online.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-09-17 10:00:11 UTC", "AssertTimestampMonotonic": "16889379", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-09-17 10:00:11 UTC", "ConditionTimestampMonotonic": "16889378", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/ 2021-09-17T10:03:45.811 INFO:teuthology.task.ansible.out:docker.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "yes", "DelegateControllers": "cpu cpuacct cpuset io blkio memory devices pids bpf-firewall bpf-devices", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "855", "ExecMainStartTimestamp": "Fri 2021-09-17 10:00:11 UTC", "ExecMainStartTimestampMonotonic": "16890326", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "docker.service", "IgnoreOnIso 2021-09-17T10:03:45.812 INFO:teuthology.task.ansible.out:late": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-09-17 10:00:11 UTC", "InactiveExitTimestampMonotonic": "16890584", "InvocationID": "43d5861b3d1a4495a55a3197fd634ec9", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127802", "LimitSIGPENDINGSoft": "127802", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "855", "MemoryAccounting": "yes", "MemoryCurrent": "137568256", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMPolicy": "continue", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace" 2021-09-17T10:03:45.814 INFO:teuthology.task.ansible.out:, "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "docker.socket sysinit.target system.slice", "Restart": "always", "RestartKillSignal": "15", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-09-17 10:00:15 UTC", "StateChangeTimestampMonotonic": "21472266", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "16", "TasksMax": "infinity", "TimeoutAbortUSec": "infinity", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset 2021-09-17T10:03:45.816 INFO:teuthology.task.ansible.out:": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "Wants": "containerd.service network-online.target", "WatchdogSignal": "6", "WatchdogTimestampMon 2021-09-17T10:03:45.816 INFO:teuthology.task.ansible.out:otonic": "0", "WatchdogUSec": "0"}} 2021-09-17T10:03:45.899 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-09-17T10:03:45.900 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-09-17T10:03:45.902 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-09-17T10:03:45.903 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-17T10:03:45.916 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Friday 17 September 2021 10:03:45 +0000 (0:00:01.668) 0:03:03.284 ****** 2021-09-17T10:03:46.916 INFO:teuthology.task.ansible.out:ok: [smithi018.front.sepia.ceph.com] 2021-09-17T10:03:46.937 INFO:teuthology.task.ansible.out:ok: [smithi050.front.sepia.ceph.com] 2021-09-17T10:03:46.956 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Friday 17 September 2021 10:03:46 +0000 (0:00:01.039) 0:03:04.324 ****** 2021-09-17T10:03:47.139 INFO:teuthology.task.ansible.out:changed: [smithi018.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-09-17T10:03:47.171 INFO:teuthology.task.ansible.out:changed: [smithi050.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-09-17T10:03:47.173 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi018.front.sepia.ceph.com : ok=131 changed=31 unreachable=0 failed=0 skipped=189 rescued=0 ignored=0 2021-09-17T10:03:47.175 INFO:teuthology.task.ansible.out:smithi050.front.sepia.ceph.com : ok=131 changed=31 unreachable=0 failed=0 skipped=189 rescued=0 ignored=0 Friday 17 September 2021 10:03:47 +0000 (0:00:00.217) 0:03:04.541 ****** =============================================================================== Create all admin users with sudo access. ------------------------------- 24.21s testnode : Install packages -------------------------------------------- 21.98s users : Update authorized_keys using the keys repo --------------------- 20.57s Remove revoked users ---------------------------------------------------- 9.51s testnode : Zap all non-root disks --------------------------------------- 8.04s testnode : Ensure packages are not present. ----------------------------- 7.87s container-host : Install registries-conf-ctl ---------------------------- 4.03s common : Update apt cache ----------------------------------------------- 3.38s testnode : Install apt keys --------------------------------------------- 2.36s Gathering Facts --------------------------------------------------------- 2.36s testnode : Update apt cache. -------------------------------------------- 2.28s testnode : Install packages via pip ------------------------------------- 2.10s testnode : Create logical volume(s) ------------------------------------- 2.07s container-host : Restart docker service --------------------------------- 1.67s users : Update authorized_keys for each user with literal keys ---------- 1.39s Gathering Facts --------------------------------------------------------- 1.22s common : Mask sleep units ----------------------------------------------- 1.21s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.08s Gathering Facts --------------------------------------------------------- 1.04s Gathering Facts --------------------------------------------------------- 0.94s 2021-09-17T10:03:47.538 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi018.front.sepia.ceph.com'), Remote(name='ubuntu@smithi050.front.sepia.ceph.com')] 2021-09-17T10:03:47.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2021-09-17T10:03:47.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-17T10:03:47.590 DEBUG:teuthology.orchestra.run.smithi018:> true 2021-09-17T10:03:47.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2021-09-17T10:03:47.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-17T10:03:47.759 DEBUG:teuthology.orchestra.run.smithi050:> true 2021-09-17T10:03:47.885 INFO:teuthology.run_tasks:Running task clock... 2021-09-17T10:03:47.896 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-09-17T10:03:47.896 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-09-17T10:03:47.899 DEBUG:teuthology.orchestra.run.smithi018:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-09-17T10:03:47.936 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:47 ntpd[7997]: ntpd 4.2.8p12@1.3728-o (1): Starting 2021-09-17T10:03:47.936 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:47 ntpd[7997]: Command line: ntpd -gq 2021-09-17T10:03:47.937 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:47 ntpd[7997]: proto: precision = 0.242 usec (-22) 2021-09-17T10:03:47.937 INFO:teuthology.orchestra.run.smithi018.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-09-17T10:03:47.938 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:47 ntpd[7997]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-09-17T10:03:47.938 INFO:teuthology.orchestra.run.smithi018.stderr:restrict ::: KOD does nothing without LIMITED. 2021-09-17T10:03:47.940 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:47 ntpd[7997]: restrict ::: KOD does nothing without LIMITED. 2021-09-17T10:03:47.940 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:47 ntpd[7997]: Listen and drop on 0 v6wildcard [::]:123 2021-09-17T10:03:47.940 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:47 ntpd[7997]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-09-17T10:03:47.940 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:47 ntpd[7997]: Listen normally on 2 lo 127.0.0.1:123 2021-09-17T10:03:47.941 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:47 ntpd[7997]: Listen normally on 3 ens1f0 172.21.15.18:123 2021-09-17T10:03:47.941 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:47 ntpd[7997]: Listen normally on 4 lo [::1]:123 2021-09-17T10:03:47.941 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:47 ntpd[7997]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1362%3]:123 2021-09-17T10:03:47.941 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:47 ntpd[7997]: Listening on routing socket on fd #22 for interface updates 2021-09-17T10:03:48.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:48.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:48.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:48.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:48.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:48.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:48.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:48.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:48.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:48.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:48.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:48.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:48.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:48.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:48.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:48.942 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:48.942 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:48.942 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:48 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:49.937 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:49 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:49.937 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:49 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:49.937 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:49 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:49.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:49 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:49.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:49 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:49.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:49 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:50.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:50.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:50.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:50.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:50.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:50.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:50.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:50.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:50.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:50.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:50.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:50.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:50.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:50.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:50.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:50.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:50.942 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:50.942 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:50 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:51.937 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:51 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:51.937 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:51 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:51.937 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:51 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:51.937 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:51 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:51.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:51 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:51.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:51 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:52.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:52.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:52.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:52.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:52.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:52.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:52.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:52.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:52.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:52.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:52.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:52.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:52.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:52.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:52.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:52.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:52.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:52.942 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:52 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:53.937 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:53 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:53.937 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:53 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:53.937 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:53 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:53.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:53 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:53.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:53 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:53.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:53 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:54.937 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:54 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:54.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:54 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:54.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:54 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:54.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:54 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:54.938 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:54 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:54.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:54 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:54.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:54 ntpd[7997]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:54.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:54 ntpd[7997]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:54.939 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:54 ntpd[7997]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:54.940 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:54 ntpd[7997]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:54.940 INFO:teuthology.orchestra.run.smithi018.stdout:17 Sep 10:03:54 ntpd[7997]: ntpd: time slew -0.000355 s 2021-09-17T10:03:54.940 INFO:teuthology.orchestra.run.smithi018.stdout:ntpd: time slew -0.000355s 2021-09-17T10:03:54.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:54 ntpd[7997]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-09-17T10:03:54.941 INFO:teuthology.orchestra.run.smithi018.stderr:17 Sep 10:03:54 ntpd[7997]: can't open /var/log/ntpstats/loopstats.20210917: Permission denied 2021-09-17T10:03:55.014 INFO:teuthology.orchestra.run.smithi018.stdout: remote refid st t when poll reach delay offset jitter 2021-09-17T10:03:55.014 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================== 2021-09-17T10:03:55.015 INFO:teuthology.orchestra.run.smithi018.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-09-17T10:03:55.015 INFO:teuthology.orchestra.run.smithi018.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-09-17T10:03:55.015 INFO:teuthology.orchestra.run.smithi018.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-09-17T10:03:55.015 INFO:teuthology.orchestra.run.smithi018.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-09-17T10:03:55.016 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-09-17T10:03:55.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-09-17T10:03:55.055 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:03:55 ntpd[7955]: ntpd 4.2.8p12@1.3728-o (1): Starting 2021-09-17T10:03:55.056 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:03:55 ntpd[7955]: Command line: ntpd -gq 2021-09-17T10:03:55.056 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:03:55 ntpd[7955]: proto: precision = 0.098 usec (-23) 2021-09-17T10:03:55.057 INFO:teuthology.orchestra.run.smithi050.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-09-17T10:03:55.057 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:03:55 ntpd[7955]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-09-17T10:03:55.058 INFO:teuthology.orchestra.run.smithi050.stderr:restrict ::: KOD does nothing without LIMITED. 2021-09-17T10:03:55.059 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:03:55 ntpd[7955]: restrict ::: KOD does nothing without LIMITED. 2021-09-17T10:03:55.059 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:03:55 ntpd[7955]: Listen and drop on 0 v6wildcard [::]:123 2021-09-17T10:03:55.060 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:03:55 ntpd[7955]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-09-17T10:03:55.060 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:03:55 ntpd[7955]: Listen normally on 2 lo 127.0.0.1:123 2021-09-17T10:03:55.060 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:03:55 ntpd[7955]: Listen normally on 3 ens1f0 172.21.15.50:123 2021-09-17T10:03:55.060 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:03:55 ntpd[7955]: Listen normally on 4 lo [::1]:123 2021-09-17T10:03:55.060 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:03:55 ntpd[7955]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1584%3]:123 2021-09-17T10:03:55.061 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:03:55 ntpd[7955]: Listening on routing socket on fd #22 for interface updates 2021-09-17T10:03:56.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:56 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:56.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:56 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:56.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:56 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:56.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:56 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:56.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:56 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:56.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:56 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:56.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:56 ntpd[7955]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:56.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:56 ntpd[7955]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:56.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:56 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:56.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:56 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:56.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:56 ntpd[7955]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:56.060 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:56 ntpd[7955]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:57.056 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:57 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:57.056 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:57 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:57 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:57 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:57 ntpd[7955]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:57.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:57 ntpd[7955]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:58.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:58 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:58.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:58 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:58.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:58 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:58 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:58 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:58 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:58.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:58 ntpd[7955]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:58.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:58 ntpd[7955]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:58.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:58 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:58.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:58 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:58.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:58 ntpd[7955]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:58.060 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:58 ntpd[7955]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:03:59.056 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:59 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:59 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:59 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:03:59.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:59 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:03:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:59 ntpd[7955]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:03:59.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:03:59 ntpd[7955]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:04:00.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:04:00.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:04:00.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:04:00.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:04:00.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:04:00.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:04:00.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:04:00.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:04:00.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:04:00.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:04:00.060 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:04:00.060 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:04:00.060 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:04:00.060 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:04:00.060 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:04:00.061 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:04:00.061 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:04:00.061 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:00 ntpd[7955]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:04:01.056 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:01 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:04:01.056 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:01 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:04:01.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:01 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:04:01.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:01 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:04:01.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:01 ntpd[7955]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:04:01.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:01 ntpd[7955]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:04:02.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:02 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:04:02.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:02 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:04:02.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:02 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:04:02.057 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:02 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:04:02.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:02 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:04:02.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:02 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:04:02.058 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:02 ntpd[7955]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-09-17T10:04:02.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:02 ntpd[7955]: can't open /var/log/ntpstats/rawstats.20210917: Permission denied 2021-09-17T10:04:02.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:02 ntpd[7955]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-09-17T10:04:02.059 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:02 ntpd[7955]: can't open /var/log/ntpstats/peerstats.20210917: Permission denied 2021-09-17T10:04:02.060 INFO:teuthology.orchestra.run.smithi050.stdout:17 Sep 10:04:02 ntpd[7955]: ntpd: time slew -0.000469 s 2021-09-17T10:04:02.060 INFO:teuthology.orchestra.run.smithi050.stdout:ntpd: time slew -0.000469s 2021-09-17T10:04:02.060 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:02 ntpd[7955]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-09-17T10:04:02.061 INFO:teuthology.orchestra.run.smithi050.stderr:17 Sep 10:04:02 ntpd[7955]: can't open /var/log/ntpstats/loopstats.20210917: Permission denied 2021-09-17T10:04:02.122 INFO:teuthology.orchestra.run.smithi050.stdout: remote refid st t when poll reach delay offset jitter 2021-09-17T10:04:02.123 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================== 2021-09-17T10:04:02.123 INFO:teuthology.orchestra.run.smithi050.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-09-17T10:04:02.123 INFO:teuthology.orchestra.run.smithi050.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-09-17T10:04:02.123 INFO:teuthology.orchestra.run.smithi050.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-09-17T10:04:02.124 INFO:teuthology.orchestra.run.smithi050.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-09-17T10:04:02.125 INFO:teuthology.run_tasks:Running task install... 2021-09-17T10:04:02.138 DEBUG:teuthology.task.install:project ceph 2021-09-17T10:04:02.138 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3'}} 2021-09-17T10:04:02.139 DEBUG:teuthology.task.install:config {'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3'} 2021-09-17T10:04:02.139 INFO:teuthology.task.install:Using flavor: basic 2021-09-17T10:04:02.146 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-09-17T10:04:02.147 INFO:teuthology.task.install:extra packages: [] 2021-09-17T10:04:02.147 DEBUG:teuthology.orchestra.run.smithi018:> sudo lsb_release -is 2021-09-17T10:04:02.197 INFO:teuthology.orchestra.run.smithi018.stdout:Ubuntu 2021-09-17T10:04:02.203 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-09-17T10:04:02.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo lsb_release -is 2021-09-17T10:04:02.204 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-key list | grep Ceph 2021-09-17T10:04:02.255 INFO:teuthology.orchestra.run.smithi050.stdout:Ubuntu 2021-09-17T10:04:02.260 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-09-17T10:04:02.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-key list | grep Ceph 2021-09-17T10:04:02.298 INFO:teuthology.orchestra.run.smithi018.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-09-17T10:04:02.360 INFO:teuthology.orchestra.run.smithi018.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-09-17T10:04:02.361 INFO:teuthology.orchestra.run.smithi018.stdout:uid [ unknown] Ceph.com (release key) 2021-09-17T10:04:02.362 DEBUG:teuthology.orchestra.run.smithi018:> sudo lsb_release -is 2021-09-17T10:04:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-09-17T10:04:02.413 INFO:teuthology.orchestra.run.smithi018.stdout:Ubuntu 2021-09-17T10:04:02.418 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-09-17T10:04:02.419 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-09-17T10:04:02.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f7edf77e75bbfd0d8d41a561025fbfc76479a1a3 2021-09-17T10:04:02.424 INFO:teuthology.orchestra.run.smithi050.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-09-17T10:04:02.424 INFO:teuthology.orchestra.run.smithi050.stdout:uid [ unknown] Ceph.com (release key) 2021-09-17T10:04:02.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo lsb_release -is 2021-09-17T10:04:02.477 INFO:teuthology.orchestra.run.smithi050.stdout:Ubuntu 2021-09-17T10:04:02.483 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-09-17T10:04:02.483 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-09-17T10:04:02.483 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f7edf77e75bbfd0d8d41a561025fbfc76479a1a3 2021-09-17T10:04:02.571 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default/ 2021-09-17T10:04:02.571 INFO:teuthology.task.install.deb:Package version is 17.0.0-7741-gf7edf77e-1focal 2021-09-17T10:04:02.669 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default/ 2021-09-17T10:04:02.669 INFO:teuthology.task.install.deb:Package version is 17.0.0-7741-gf7edf77e-1focal 2021-09-17T10:04:02.849 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:04:02.849 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-09-17T10:04:02.866 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get update 2021-09-17T10:04:02.941 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:04:02.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-09-17T10:04:02.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get update 2021-09-17T10:04:03.109 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-09-17T10:04:03.153 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-09-17T10:04:03.224 INFO:teuthology.orchestra.run.smithi018.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal InRelease 2021-09-17T10:04:03.261 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-09-17T10:04:03.285 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal Release [7,662 B] 2021-09-17T10:04:03.331 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2021-09-17T10:04:03.344 INFO:teuthology.orchestra.run.smithi018.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal Release.gpg 2021-09-17T10:04:03.379 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:7 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-09-17T10:04:03.405 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 Packages [17.3 kB] 2021-09-17T10:04:03.424 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-09-17T10:04:03.489 INFO:teuthology.orchestra.run.smithi050.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal InRelease 2021-09-17T10:04:03.512 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-09-17T10:04:03.540 INFO:teuthology.orchestra.run.smithi018.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main i386 Packages [3,347 B] 2021-09-17T10:04:03.553 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal Release [7,662 B] 2021-09-17T10:04:03.643 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-09-17T10:04:03.653 INFO:teuthology.orchestra.run.smithi050.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal Release.gpg 2021-09-17T10:04:03.749 INFO:teuthology.orchestra.run.smithi050.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main i386 Packages [3,347 B] 2021-09-17T10:04:03.802 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 28.3 kB in 1s (36.7 kB/s) 2021-09-17T10:04:03.832 INFO:teuthology.orchestra.run.smithi050.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 Packages [17.3 kB] 2021-09-17T10:04:03.999 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 28.3 kB in 1s (34.1 kB/s) 2021-09-17T10:04:04.601 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2021-09-17T10:04:04.625 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-7741-gf7edf77e-1focal cephadm=17.0.0-7741-gf7edf77e-1focal ceph-mds=17.0.0-7741-gf7edf77e-1focal ceph-mgr=17.0.0-7741-gf7edf77e-1focal ceph-common=17.0.0-7741-gf7edf77e-1focal ceph-fuse=17.0.0-7741-gf7edf77e-1focal ceph-test=17.0.0-7741-gf7edf77e-1focal radosgw=17.0.0-7741-gf7edf77e-1focal python3-rados=17.0.0-7741-gf7edf77e-1focal python3-rgw=17.0.0-7741-gf7edf77e-1focal python3-cephfs=17.0.0-7741-gf7edf77e-1focal python3-rbd=17.0.0-7741-gf7edf77e-1focal libcephfs2=17.0.0-7741-gf7edf77e-1focal libcephfs-dev=17.0.0-7741-gf7edf77e-1focal librados2=17.0.0-7741-gf7edf77e-1focal librbd1=17.0.0-7741-gf7edf77e-1focal rbd-fuse=17.0.0-7741-gf7edf77e-1focal 2021-09-17T10:04:04.693 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2021-09-17T10:04:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2021-09-17T10:04:04.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-7741-gf7edf77e-1focal cephadm=17.0.0-7741-gf7edf77e-1focal ceph-mds=17.0.0-7741-gf7edf77e-1focal ceph-mgr=17.0.0-7741-gf7edf77e-1focal ceph-common=17.0.0-7741-gf7edf77e-1focal ceph-fuse=17.0.0-7741-gf7edf77e-1focal ceph-test=17.0.0-7741-gf7edf77e-1focal radosgw=17.0.0-7741-gf7edf77e-1focal python3-rados=17.0.0-7741-gf7edf77e-1focal python3-rgw=17.0.0-7741-gf7edf77e-1focal python3-cephfs=17.0.0-7741-gf7edf77e-1focal python3-rbd=17.0.0-7741-gf7edf77e-1focal libcephfs2=17.0.0-7741-gf7edf77e-1focal libcephfs-dev=17.0.0-7741-gf7edf77e-1focal librados2=17.0.0-7741-gf7edf77e-1focal librbd1=17.0.0-7741-gf7edf77e-1focal rbd-fuse=17.0.0-7741-gf7edf77e-1focal 2021-09-17T10:04:04.843 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2021-09-17T10:04:04.844 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2021-09-17T10:04:04.892 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2021-09-17T10:04:04.976 INFO:teuthology.orchestra.run.smithi018.stdout:The following packages were automatically installed and are no longer required: 2021-09-17T10:04:04.978 INFO:teuthology.orchestra.run.smithi018.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2021-09-17T10:04:04.979 INFO:teuthology.orchestra.run.smithi018.stdout:Use 'sudo apt autoremove' to remove them. 2021-09-17T10:04:04.979 INFO:teuthology.orchestra.run.smithi018.stdout:The following additional packages will be installed: 2021-09-17T10:04:04.979 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-09-17T10:04:04.980 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd ceph-volume jq 2021-09-17T10:04:04.980 INFO:teuthology.orchestra.run.smithi018.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2021-09-17T10:04:04.980 INFO:teuthology.orchestra.run.smithi018.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2021-09-17T10:04:04.980 INFO:teuthology.orchestra.run.smithi018.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libxslt1.1 nvme-cli 2021-09-17T10:04:04.981 INFO:teuthology.orchestra.run.smithi018.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2021-09-17T10:04:04.981 INFO:teuthology.orchestra.run.smithi018.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2021-09-17T10:04:04.981 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2021-09-17T10:04:04.982 INFO:teuthology.orchestra.run.smithi018.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2021-09-17T10:04:04.982 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-09-17T10:04:04.983 INFO:teuthology.orchestra.run.smithi018.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2021-09-17T10:04:04.983 INFO:teuthology.orchestra.run.smithi018.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2021-09-17T10:04:04.983 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2021-09-17T10:04:04.984 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2021-09-17T10:04:04.984 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2021-09-17T10:04:04.985 INFO:teuthology.orchestra.run.smithi018.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-09-17T10:04:04.985 INFO:teuthology.orchestra.run.smithi018.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2021-09-17T10:04:04.985 INFO:teuthology.orchestra.run.smithi018.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2021-09-17T10:04:04.986 INFO:teuthology.orchestra.run.smithi018.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2021-09-17T10:04:04.986 INFO:teuthology.orchestra.run.smithi018.stdout:Suggested packages: 2021-09-17T10:04:04.986 INFO:teuthology.orchestra.run.smithi018.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2021-09-17T10:04:04.987 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2021-09-17T10:04:04.987 INFO:teuthology.orchestra.run.smithi018.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2021-09-17T10:04:04.988 INFO:teuthology.orchestra.run.smithi018.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2021-09-17T10:04:04.988 INFO:teuthology.orchestra.run.smithi018.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2021-09-17T10:04:04.988 INFO:teuthology.orchestra.run.smithi018.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2021-09-17T10:04:04.989 INFO:teuthology.orchestra.run.smithi018.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2021-09-17T10:04:04.989 INFO:teuthology.orchestra.run.smithi018.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2021-09-17T10:04:04.990 INFO:teuthology.orchestra.run.smithi018.stdout:Recommended packages: 2021-09-17T10:04:04.990 INFO:teuthology.orchestra.run.smithi018.stdout: btrfs-tools 2021-09-17T10:04:05.061 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2021-09-17T10:04:05.062 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2021-09-17T10:04:05.076 INFO:teuthology.orchestra.run.smithi018.stdout:The following NEW packages will be installed: 2021-09-17T10:04:05.077 INFO:teuthology.orchestra.run.smithi018.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-09-17T10:04:05.077 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-09-17T10:04:05.078 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2021-09-17T10:04:05.078 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs-dev libcephfs2 libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2021-09-17T10:04:05.078 INFO:teuthology.orchestra.run.smithi018.stdout: libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 2021-09-17T10:04:05.079 INFO:teuthology.orchestra.run.smithi018.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libxslt1.1 nvme-cli 2021-09-17T10:04:05.079 INFO:teuthology.orchestra.run.smithi018.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2021-09-17T10:04:05.080 INFO:teuthology.orchestra.run.smithi018.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2021-09-17T10:04:05.081 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2021-09-17T10:04:05.081 INFO:teuthology.orchestra.run.smithi018.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2021-09-17T10:04:05.082 INFO:teuthology.orchestra.run.smithi018.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2021-09-17T10:04:05.082 INFO:teuthology.orchestra.run.smithi018.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2021-09-17T10:04:05.083 INFO:teuthology.orchestra.run.smithi018.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2021-09-17T10:04:05.083 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2021-09-17T10:04:05.084 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2021-09-17T10:04:05.084 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2021-09-17T10:04:05.084 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2021-09-17T10:04:05.085 INFO:teuthology.orchestra.run.smithi018.stdout: python3-simplejson python3-singledispatch python3-sklearn 2021-09-17T10:04:05.085 INFO:teuthology.orchestra.run.smithi018.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2021-09-17T10:04:05.086 INFO:teuthology.orchestra.run.smithi018.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2021-09-17T10:04:05.086 INFO:teuthology.orchestra.run.smithi018.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-09-17T10:04:05.087 INFO:teuthology.orchestra.run.smithi018.stdout:The following packages will be upgraded: 2021-09-17T10:04:05.087 INFO:teuthology.orchestra.run.smithi018.stdout: librados2 librbd1 2021-09-17T10:04:05.207 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages were automatically installed and are no longer required: 2021-09-17T10:04:05.209 INFO:teuthology.orchestra.run.smithi050.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2021-09-17T10:04:05.209 INFO:teuthology.orchestra.run.smithi050.stdout:Use 'sudo apt autoremove' to remove them. 2021-09-17T10:04:05.210 INFO:teuthology.orchestra.run.smithi050.stdout:The following additional packages will be installed: 2021-09-17T10:04:05.210 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-09-17T10:04:05.210 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd ceph-volume jq 2021-09-17T10:04:05.210 INFO:teuthology.orchestra.run.smithi050.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2021-09-17T10:04:05.211 INFO:teuthology.orchestra.run.smithi050.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2021-09-17T10:04:05.211 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libxslt1.1 nvme-cli 2021-09-17T10:04:05.211 INFO:teuthology.orchestra.run.smithi050.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2021-09-17T10:04:05.212 INFO:teuthology.orchestra.run.smithi050.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2021-09-17T10:04:05.212 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2021-09-17T10:04:05.212 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2021-09-17T10:04:05.212 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-09-17T10:04:05.213 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2021-09-17T10:04:05.213 INFO:teuthology.orchestra.run.smithi050.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2021-09-17T10:04:05.213 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2021-09-17T10:04:05.213 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2021-09-17T10:04:05.214 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2021-09-17T10:04:05.214 INFO:teuthology.orchestra.run.smithi050.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-09-17T10:04:05.214 INFO:teuthology.orchestra.run.smithi050.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2021-09-17T10:04:05.214 INFO:teuthology.orchestra.run.smithi050.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2021-09-17T10:04:05.214 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2021-09-17T10:04:05.215 INFO:teuthology.orchestra.run.smithi050.stdout:Suggested packages: 2021-09-17T10:04:05.215 INFO:teuthology.orchestra.run.smithi050.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2021-09-17T10:04:05.215 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2021-09-17T10:04:05.216 INFO:teuthology.orchestra.run.smithi050.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2021-09-17T10:04:05.216 INFO:teuthology.orchestra.run.smithi050.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2021-09-17T10:04:05.216 INFO:teuthology.orchestra.run.smithi050.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2021-09-17T10:04:05.216 INFO:teuthology.orchestra.run.smithi050.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2021-09-17T10:04:05.217 INFO:teuthology.orchestra.run.smithi050.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2021-09-17T10:04:05.217 INFO:teuthology.orchestra.run.smithi050.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2021-09-17T10:04:05.217 INFO:teuthology.orchestra.run.smithi050.stdout:Recommended packages: 2021-09-17T10:04:05.218 INFO:teuthology.orchestra.run.smithi050.stdout: btrfs-tools 2021-09-17T10:04:05.273 INFO:teuthology.orchestra.run.smithi018.stdout:2 upgraded, 99 newly installed, 0 to remove and 167 not upgraded. 2021-09-17T10:04:05.273 INFO:teuthology.orchestra.run.smithi018.stdout:Need to get 177 MB of archives. 2021-09-17T10:04:05.274 INFO:teuthology.orchestra.run.smithi018.stdout:After this operation, 829 MB of additional disk space will be used. 2021-09-17T10:04:05.274 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2021-09-17T10:04:05.305 INFO:teuthology.orchestra.run.smithi050.stdout:The following NEW packages will be installed: 2021-09-17T10:04:05.305 INFO:teuthology.orchestra.run.smithi050.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-09-17T10:04:05.306 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-09-17T10:04:05.306 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2021-09-17T10:04:05.306 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-dev libcephfs2 libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2021-09-17T10:04:05.307 INFO:teuthology.orchestra.run.smithi050.stdout: libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 2021-09-17T10:04:05.307 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libxslt1.1 nvme-cli 2021-09-17T10:04:05.307 INFO:teuthology.orchestra.run.smithi050.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2021-09-17T10:04:05.308 INFO:teuthology.orchestra.run.smithi050.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2021-09-17T10:04:05.308 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2021-09-17T10:04:05.308 INFO:teuthology.orchestra.run.smithi050.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2021-09-17T10:04:05.308 INFO:teuthology.orchestra.run.smithi050.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2021-09-17T10:04:05.309 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2021-09-17T10:04:05.309 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2021-09-17T10:04:05.309 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2021-09-17T10:04:05.309 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2021-09-17T10:04:05.310 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2021-09-17T10:04:05.310 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2021-09-17T10:04:05.310 INFO:teuthology.orchestra.run.smithi050.stdout: python3-simplejson python3-singledispatch python3-sklearn 2021-09-17T10:04:05.311 INFO:teuthology.orchestra.run.smithi050.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2021-09-17T10:04:05.311 INFO:teuthology.orchestra.run.smithi050.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2021-09-17T10:04:05.311 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-09-17T10:04:05.312 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages will be upgraded: 2021-09-17T10:04:05.312 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 librbd1 2021-09-17T10:04:05.451 INFO:teuthology.orchestra.run.smithi018.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-7741-gf7edf77e-1focal [3,472 kB] 2021-09-17T10:04:05.513 INFO:teuthology.orchestra.run.smithi050.stdout:2 upgraded, 99 newly installed, 0 to remove and 167 not upgraded. 2021-09-17T10:04:05.514 INFO:teuthology.orchestra.run.smithi050.stdout:Need to get 177 MB of archives. 2021-09-17T10:04:05.514 INFO:teuthology.orchestra.run.smithi050.stdout:After this operation, 829 MB of additional disk space will be used. 2021-09-17T10:04:05.514 INFO:teuthology.orchestra.run.smithi050.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2021-09-17T10:04:05.673 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2021-09-17T10:04:05.683 INFO:teuthology.orchestra.run.smithi018.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2021-09-17T10:04:05.695 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2021-09-17T10:04:05.752 INFO:teuthology.orchestra.run.smithi050.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-7741-gf7edf77e-1focal [3,472 kB] 2021-09-17T10:04:05.762 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2021-09-17T10:04:05.873 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2021-09-17T10:04:05.893 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2021-09-17T10:04:05.898 INFO:teuthology.orchestra.run.smithi018.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2021-09-17T10:04:05.920 INFO:teuthology.orchestra.run.smithi018.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2021-09-17T10:04:05.924 INFO:teuthology.orchestra.run.smithi018.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2021-09-17T10:04:05.936 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2021-09-17T10:04:05.937 INFO:teuthology.orchestra.run.smithi050.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2021-09-17T10:04:05.941 INFO:teuthology.orchestra.run.smithi018.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2021-09-17T10:04:05.957 INFO:teuthology.orchestra.run.smithi018.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2021-09-17T10:04:05.958 INFO:teuthology.orchestra.run.smithi018.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2021-09-17T10:04:05.960 INFO:teuthology.orchestra.run.smithi018.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2021-09-17T10:04:05.965 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2021-09-17T10:04:06.024 INFO:teuthology.orchestra.run.smithi050.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2021-09-17T10:04:06.027 INFO:teuthology.orchestra.run.smithi018.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2021-09-17T10:04:06.030 INFO:teuthology.orchestra.run.smithi018.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2021-09-17T10:04:06.031 INFO:teuthology.orchestra.run.smithi018.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2021-09-17T10:04:06.032 INFO:teuthology.orchestra.run.smithi018.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2021-09-17T10:04:06.038 INFO:teuthology.orchestra.run.smithi018.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2021-09-17T10:04:06.115 INFO:teuthology.orchestra.run.smithi018.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2021-09-17T10:04:06.120 INFO:teuthology.orchestra.run.smithi018.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2021-09-17T10:04:06.124 INFO:teuthology.orchestra.run.smithi018.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2021-09-17T10:04:06.125 INFO:teuthology.orchestra.run.smithi018.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2021-09-17T10:04:06.151 INFO:teuthology.orchestra.run.smithi018.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2021-09-17T10:04:06.151 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2021-09-17T10:04:06.173 INFO:teuthology.orchestra.run.smithi050.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2021-09-17T10:04:06.180 INFO:teuthology.orchestra.run.smithi050.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2021-09-17T10:04:06.207 INFO:teuthology.orchestra.run.smithi018.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2021-09-17T10:04:06.208 INFO:teuthology.orchestra.run.smithi018.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2021-09-17T10:04:06.210 INFO:teuthology.orchestra.run.smithi018.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2021-09-17T10:04:06.211 INFO:teuthology.orchestra.run.smithi050.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2021-09-17T10:04:06.215 INFO:teuthology.orchestra.run.smithi018.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2021-09-17T10:04:06.216 INFO:teuthology.orchestra.run.smithi050.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2021-09-17T10:04:06.221 INFO:teuthology.orchestra.run.smithi050.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2021-09-17T10:04:06.226 INFO:teuthology.orchestra.run.smithi018.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2021-09-17T10:04:06.239 INFO:teuthology.orchestra.run.smithi050.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2021-09-17T10:04:06.241 INFO:teuthology.orchestra.run.smithi050.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2021-09-17T10:04:06.244 INFO:teuthology.orchestra.run.smithi050.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2021-09-17T10:04:06.247 INFO:teuthology.orchestra.run.smithi050.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2021-09-17T10:04:06.301 INFO:teuthology.orchestra.run.smithi018.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2021-09-17T10:04:06.301 INFO:teuthology.orchestra.run.smithi018.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2021-09-17T10:04:06.306 INFO:teuthology.orchestra.run.smithi018.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2021-09-17T10:04:06.312 INFO:teuthology.orchestra.run.smithi018.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2021-09-17T10:04:06.320 INFO:teuthology.orchestra.run.smithi018.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2021-09-17T10:04:06.342 INFO:teuthology.orchestra.run.smithi050.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2021-09-17T10:04:06.343 INFO:teuthology.orchestra.run.smithi050.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2021-09-17T10:04:06.343 INFO:teuthology.orchestra.run.smithi050.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2021-09-17T10:04:06.346 INFO:teuthology.orchestra.run.smithi050.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2021-09-17T10:04:06.348 INFO:teuthology.orchestra.run.smithi050.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2021-09-17T10:04:06.351 INFO:teuthology.orchestra.run.smithi050.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2021-09-17T10:04:06.355 INFO:teuthology.orchestra.run.smithi050.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2021-09-17T10:04:06.355 INFO:teuthology.orchestra.run.smithi050.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2021-09-17T10:04:06.376 INFO:teuthology.orchestra.run.smithi050.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2021-09-17T10:04:06.399 INFO:teuthology.orchestra.run.smithi018.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2021-09-17T10:04:06.399 INFO:teuthology.orchestra.run.smithi018.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2021-09-17T10:04:06.446 INFO:teuthology.orchestra.run.smithi050.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2021-09-17T10:04:06.447 INFO:teuthology.orchestra.run.smithi050.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2021-09-17T10:04:06.449 INFO:teuthology.orchestra.run.smithi050.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2021-09-17T10:04:06.458 INFO:teuthology.orchestra.run.smithi018.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-7741-gf7edf77e-1focal [3,514 kB] 2021-09-17T10:04:06.546 INFO:teuthology.orchestra.run.smithi050.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2021-09-17T10:04:06.554 INFO:teuthology.orchestra.run.smithi050.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2021-09-17T10:04:06.557 INFO:teuthology.orchestra.run.smithi050.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2021-09-17T10:04:06.558 INFO:teuthology.orchestra.run.smithi050.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2021-09-17T10:04:06.560 INFO:teuthology.orchestra.run.smithi050.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2021-09-17T10:04:06.562 INFO:teuthology.orchestra.run.smithi050.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2021-09-17T10:04:06.566 INFO:teuthology.orchestra.run.smithi050.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2021-09-17T10:04:06.652 INFO:teuthology.orchestra.run.smithi050.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2021-09-17T10:04:06.653 INFO:teuthology.orchestra.run.smithi050.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2021-09-17T10:04:06.905 INFO:teuthology.orchestra.run.smithi018.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-7741-gf7edf77e-1focal [744 kB] 2021-09-17T10:04:06.941 INFO:teuthology.orchestra.run.smithi018.stdout:Get:40 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-7741-gf7edf77e-1focal [408 kB] 2021-09-17T10:04:06.950 INFO:teuthology.orchestra.run.smithi018.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2021-09-17T10:04:06.961 INFO:teuthology.orchestra.run.smithi018.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-7741-gf7edf77e-1focal [22.4 kB] 2021-09-17T10:04:06.962 INFO:teuthology.orchestra.run.smithi018.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-7741-gf7edf77e-1focal [200 kB] 2021-09-17T10:04:06.969 INFO:teuthology.orchestra.run.smithi018.stdout:Get:44 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-7741-gf7edf77e-1focal [31.1 kB] 2021-09-17T10:04:06.970 INFO:teuthology.orchestra.run.smithi018.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-7741-gf7edf77e-1focal [385 kB] 2021-09-17T10:04:06.986 INFO:teuthology.orchestra.run.smithi018.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-7741-gf7edf77e-1focal [3,891 kB] 2021-09-17T10:04:07.119 INFO:teuthology.orchestra.run.smithi018.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2021-09-17T10:04:07.121 INFO:teuthology.orchestra.run.smithi018.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2021-09-17T10:04:07.137 INFO:teuthology.orchestra.run.smithi018.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2021-09-17T10:04:07.137 INFO:teuthology.orchestra.run.smithi018.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2021-09-17T10:04:07.137 INFO:teuthology.orchestra.run.smithi018.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2021-09-17T10:04:07.138 INFO:teuthology.orchestra.run.smithi018.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2021-09-17T10:04:07.138 INFO:teuthology.orchestra.run.smithi018.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2021-09-17T10:04:07.139 INFO:teuthology.orchestra.run.smithi018.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2021-09-17T10:04:07.146 INFO:teuthology.orchestra.run.smithi018.stdout:Get:55 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-7741-gf7edf77e-1focal [116 kB] 2021-09-17T10:04:07.150 INFO:teuthology.orchestra.run.smithi018.stdout:Get:56 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-7741-gf7edf77e-1focal [477 kB] 2021-09-17T10:04:07.171 INFO:teuthology.orchestra.run.smithi018.stdout:Get:57 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-7741-gf7edf77e-1focal [23.3 MB] 2021-09-17T10:04:07.179 INFO:teuthology.orchestra.run.smithi018.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2021-09-17T10:04:07.203 INFO:teuthology.orchestra.run.smithi018.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2021-09-17T10:04:07.266 INFO:teuthology.orchestra.run.smithi050.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2021-09-17T10:04:07.299 INFO:teuthology.orchestra.run.smithi018.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2021-09-17T10:04:07.300 INFO:teuthology.orchestra.run.smithi018.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2021-09-17T10:04:07.305 INFO:teuthology.orchestra.run.smithi018.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2021-09-17T10:04:07.308 INFO:teuthology.orchestra.run.smithi018.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2021-09-17T10:04:07.312 INFO:teuthology.orchestra.run.smithi018.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2021-09-17T10:04:07.312 INFO:teuthology.orchestra.run.smithi018.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2021-09-17T10:04:07.315 INFO:teuthology.orchestra.run.smithi018.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2021-09-17T10:04:07.316 INFO:teuthology.orchestra.run.smithi018.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2021-09-17T10:04:07.318 INFO:teuthology.orchestra.run.smithi018.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2021-09-17T10:04:07.333 INFO:teuthology.orchestra.run.smithi050.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-7741-gf7edf77e-1focal [3,514 kB] 2021-09-17T10:04:07.351 INFO:teuthology.orchestra.run.smithi050.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2021-09-17T10:04:07.354 INFO:teuthology.orchestra.run.smithi050.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2021-09-17T10:04:07.389 INFO:teuthology.orchestra.run.smithi050.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2021-09-17T10:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2021-09-17T10:04:07.390 INFO:teuthology.orchestra.run.smithi050.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2021-09-17T10:04:07.391 INFO:teuthology.orchestra.run.smithi050.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2021-09-17T10:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2021-09-17T10:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2021-09-17T10:04:07.393 INFO:teuthology.orchestra.run.smithi050.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2021-09-17T10:04:07.395 INFO:teuthology.orchestra.run.smithi018.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2021-09-17T10:04:07.395 INFO:teuthology.orchestra.run.smithi018.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2021-09-17T10:04:07.445 INFO:teuthology.orchestra.run.smithi050.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2021-09-17T10:04:07.446 INFO:teuthology.orchestra.run.smithi050.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2021-09-17T10:04:07.480 INFO:teuthology.orchestra.run.smithi018.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2021-09-17T10:04:07.481 INFO:teuthology.orchestra.run.smithi018.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2021-09-17T10:04:07.483 INFO:teuthology.orchestra.run.smithi018.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.3 [1,382 kB] 2021-09-17T10:04:07.512 INFO:teuthology.orchestra.run.smithi018.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2021-09-17T10:04:07.512 INFO:teuthology.orchestra.run.smithi018.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2021-09-17T10:04:07.513 INFO:teuthology.orchestra.run.smithi018.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2021-09-17T10:04:07.514 INFO:teuthology.orchestra.run.smithi018.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2021-09-17T10:04:07.515 INFO:teuthology.orchestra.run.smithi018.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-py all 1.8.1-1 [65.4 kB] 2021-09-17T10:04:07.532 INFO:teuthology.orchestra.run.smithi050.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2021-09-17T10:04:07.537 INFO:teuthology.orchestra.run.smithi050.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2021-09-17T10:04:07.540 INFO:teuthology.orchestra.run.smithi050.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2021-09-17T10:04:07.543 INFO:teuthology.orchestra.run.smithi050.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2021-09-17T10:04:07.545 INFO:teuthology.orchestra.run.smithi050.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2021-09-17T10:04:07.549 INFO:teuthology.orchestra.run.smithi050.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2021-09-17T10:04:07.550 INFO:teuthology.orchestra.run.smithi050.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2021-09-17T10:04:07.552 INFO:teuthology.orchestra.run.smithi050.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2021-09-17T10:04:07.575 INFO:teuthology.orchestra.run.smithi018.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2021-09-17T10:04:07.575 INFO:teuthology.orchestra.run.smithi018.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2021-09-17T10:04:07.620 INFO:teuthology.orchestra.run.smithi050.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2021-09-17T10:04:07.621 INFO:teuthology.orchestra.run.smithi050.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2021-09-17T10:04:07.668 INFO:teuthology.orchestra.run.smithi018.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2021-09-17T10:04:07.672 INFO:teuthology.orchestra.run.smithi018.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2021-09-17T10:04:07.715 INFO:teuthology.orchestra.run.smithi050.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2021-09-17T10:04:07.716 INFO:teuthology.orchestra.run.smithi050.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2021-09-17T10:04:07.718 INFO:teuthology.orchestra.run.smithi050.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.3 [1,382 kB] 2021-09-17T10:04:07.737 INFO:teuthology.orchestra.run.smithi050.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2021-09-17T10:04:07.738 INFO:teuthology.orchestra.run.smithi050.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2021-09-17T10:04:07.739 INFO:teuthology.orchestra.run.smithi050.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2021-09-17T10:04:07.739 INFO:teuthology.orchestra.run.smithi050.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2021-09-17T10:04:07.741 INFO:teuthology.orchestra.run.smithi050.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-py all 1.8.1-1 [65.4 kB] 2021-09-17T10:04:07.798 INFO:teuthology.orchestra.run.smithi050.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2021-09-17T10:04:07.801 INFO:teuthology.orchestra.run.smithi050.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2021-09-17T10:04:07.889 INFO:teuthology.orchestra.run.smithi050.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2021-09-17T10:04:07.892 INFO:teuthology.orchestra.run.smithi050.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2021-09-17T10:04:08.161 INFO:teuthology.orchestra.run.smithi018.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-7741-gf7edf77e-1focal [6,401 kB] 2021-09-17T10:04:08.404 INFO:teuthology.orchestra.run.smithi018.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-7741-gf7edf77e-1focal [198 kB] 2021-09-17T10:04:08.408 INFO:teuthology.orchestra.run.smithi018.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-7741-gf7edf77e-1focal [152 kB] 2021-09-17T10:04:08.411 INFO:teuthology.orchestra.run.smithi018.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-7741-gf7edf77e-1focal [1,548 kB] 2021-09-17T10:04:08.422 INFO:teuthology.orchestra.run.smithi018.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-7741-gf7edf77e-1focal [7,806 kB] 2021-09-17T10:04:08.684 INFO:teuthology.orchestra.run.smithi018.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-7741-gf7edf77e-1focal [28.4 MB] 2021-09-17T10:04:09.087 INFO:teuthology.orchestra.run.smithi050.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-7741-gf7edf77e-1focal [744 kB] 2021-09-17T10:04:09.481 INFO:teuthology.orchestra.run.smithi050.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-7741-gf7edf77e-1focal [408 kB] 2021-09-17T10:04:09.666 INFO:teuthology.orchestra.run.smithi050.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-7741-gf7edf77e-1focal [22.4 kB] 2021-09-17T10:04:09.689 INFO:teuthology.orchestra.run.smithi050.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-7741-gf7edf77e-1focal [200 kB] 2021-09-17T10:04:09.768 INFO:teuthology.orchestra.run.smithi050.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-7741-gf7edf77e-1focal [31.1 kB] 2021-09-17T10:04:09.772 INFO:teuthology.orchestra.run.smithi050.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-7741-gf7edf77e-1focal [385 kB] 2021-09-17T10:04:09.850 INFO:teuthology.orchestra.run.smithi018.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-7741-gf7edf77e-1focal [3,816 B] 2021-09-17T10:04:09.922 INFO:teuthology.orchestra.run.smithi018.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-7741-gf7edf77e-1focal [850 kB] 2021-09-17T10:04:09.929 INFO:teuthology.orchestra.run.smithi050.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-7741-gf7edf77e-1focal [3,891 kB] 2021-09-17T10:04:09.933 INFO:teuthology.orchestra.run.smithi018.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-7741-gf7edf77e-1focal [2,353 kB] 2021-09-17T10:04:09.952 INFO:teuthology.orchestra.run.smithi018.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-7741-gf7edf77e-1focal [68.1 kB] 2021-09-17T10:04:09.953 INFO:teuthology.orchestra.run.smithi018.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.0.0-7741-gf7edf77e-1focal [86.6 kB] 2021-09-17T10:04:09.954 INFO:teuthology.orchestra.run.smithi018.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-7741-gf7edf77e-1focal [1,593 kB] 2021-09-17T10:04:09.978 INFO:teuthology.orchestra.run.smithi018.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-7741-gf7edf77e-1focal [7,873 kB] 2021-09-17T10:04:10.281 INFO:teuthology.orchestra.run.smithi018.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-7741-gf7edf77e-1focal [3,936 B] 2021-09-17T10:04:10.282 INFO:teuthology.orchestra.run.smithi018.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-7741-gf7edf77e-1focal [48.8 MB] 2021-09-17T10:04:11.774 INFO:teuthology.orchestra.run.smithi050.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-7741-gf7edf77e-1focal [116 kB] 2021-09-17T10:04:11.803 INFO:teuthology.orchestra.run.smithi050.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-7741-gf7edf77e-1focal [477 kB] 2021-09-17T10:04:12.043 INFO:teuthology.orchestra.run.smithi050.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-7741-gf7edf77e-1focal [23.3 MB] 2021-09-17T10:04:12.402 INFO:teuthology.orchestra.run.smithi018.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.0.0-7741-gf7edf77e-1focal [215 kB] 2021-09-17T10:04:12.406 INFO:teuthology.orchestra.run.smithi018.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-7741-gf7edf77e-1focal [20.8 kB] 2021-09-17T10:04:12.407 INFO:teuthology.orchestra.run.smithi018.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-7741-gf7edf77e-1focal [11.8 MB] 2021-09-17T10:04:12.791 INFO:teuthology.orchestra.run.smithi018.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-7741-gf7edf77e-1focal [78.9 kB] 2021-09-17T10:04:13.385 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 177 MB in 8s (23.0 MB/s) 2021-09-17T10:04:13.575 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-09-17T10:04:13.631 INFO:teuthology.orchestra.run.smithi018.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138119 files and directories currently installed.) 2021-09-17T10:04:13.635 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2021-09-17T10:04:13.749 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2021-09-17T10:04:14.079 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../001-librbd1_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:14.235 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking librbd1 (17.0.0-7741-gf7edf77e-1focal) over (15.2.13-0ubuntu0.20.04.2) ... 2021-09-17T10:04:14.938 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../002-librados2_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:15.092 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking librados2 (17.0.0-7741-gf7edf77e-1focal) over (15.2.13-0ubuntu0.20.04.2) ... 2021-09-17T10:04:15.583 INFO:teuthology.orchestra.run.smithi050.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-7741-gf7edf77e-1focal [6,401 kB] 2021-09-17T10:04:15.775 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libcephfs2. 2021-09-17T10:04:15.790 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../003-libcephfs2_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:15.858 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libcephfs2 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:16.185 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-rados. 2021-09-17T10:04:16.204 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../004-python3-rados_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:16.276 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-rados (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:16.457 INFO:teuthology.orchestra.run.smithi050.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-7741-gf7edf77e-1focal [198 kB] 2021-09-17T10:04:16.481 INFO:teuthology.orchestra.run.smithi050.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-7741-gf7edf77e-1focal [152 kB] 2021-09-17T10:04:16.499 INFO:teuthology.orchestra.run.smithi050.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-7741-gf7edf77e-1focal [1,548 kB] 2021-09-17T10:04:16.668 INFO:teuthology.orchestra.run.smithi050.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-7741-gf7edf77e-1focal [7,806 kB] 2021-09-17T10:04:16.784 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-09-17T10:04:16.802 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:04:16.843 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-ceph-argparse (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:17.109 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-cephfs. 2021-09-17T10:04:17.128 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../006-python3-cephfs_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:17.168 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-cephfs (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:17.451 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-ceph-common. 2021-09-17T10:04:17.469 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../007-python3-ceph-common_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:04:17.532 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-ceph-common (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:17.543 INFO:teuthology.orchestra.run.smithi050.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-7741-gf7edf77e-1focal [28.4 MB] 2021-09-17T10:04:17.816 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-prettytable. 2021-09-17T10:04:17.836 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2021-09-17T10:04:17.867 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2021-09-17T10:04:18.158 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-rbd. 2021-09-17T10:04:18.177 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../009-python3-rbd_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:18.209 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-rbd (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:18.583 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2021-09-17T10:04:18.600 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2021-09-17T10:04:18.634 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-09-17T10:04:18.982 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-09-17T10:04:19.001 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2021-09-17T10:04:19.035 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2021-09-17T10:04:19.335 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-09-17T10:04:19.351 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2021-09-17T10:04:19.395 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-09-17T10:04:19.772 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package librgw2. 2021-09-17T10:04:19.789 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../013-librgw2_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:19.830 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking librgw2 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:20.088 INFO:teuthology.orchestra.run.smithi050.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-7741-gf7edf77e-1focal [3,816 B] 2021-09-17T10:04:20.088 INFO:teuthology.orchestra.run.smithi050.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-7741-gf7edf77e-1focal [850 kB] 2021-09-17T10:04:20.104 INFO:teuthology.orchestra.run.smithi050.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-7741-gf7edf77e-1focal [2,353 kB] 2021-09-17T10:04:20.224 INFO:teuthology.orchestra.run.smithi050.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-7741-gf7edf77e-1focal [68.1 kB] 2021-09-17T10:04:20.226 INFO:teuthology.orchestra.run.smithi050.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.0.0-7741-gf7edf77e-1focal [86.6 kB] 2021-09-17T10:04:20.251 INFO:teuthology.orchestra.run.smithi050.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-7741-gf7edf77e-1focal [1,593 kB] 2021-09-17T10:04:20.356 INFO:teuthology.orchestra.run.smithi050.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-7741-gf7edf77e-1focal [7,873 kB] 2021-09-17T10:04:20.470 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-rgw. 2021-09-17T10:04:20.488 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../014-python3-rgw_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:20.527 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-rgw (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:20.884 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libleveldb1d:amd64. 2021-09-17T10:04:20.901 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../015-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2021-09-17T10:04:20.929 INFO:teuthology.orchestra.run.smithi050.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-7741-gf7edf77e-1focal [3,936 B] 2021-09-17T10:04:20.929 INFO:teuthology.orchestra.run.smithi050.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-7741-gf7edf77e-1focal [48.8 MB] 2021-09-17T10:04:20.935 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-09-17T10:04:21.302 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package liboath0. 2021-09-17T10:04:21.320 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../016-liboath0_2.6.1-1.3_amd64.deb ... 2021-09-17T10:04:21.352 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2021-09-17T10:04:21.618 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libradosstriper1. 2021-09-17T10:04:21.634 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../017-libradosstriper1_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:21.688 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libradosstriper1 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:22.173 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-common. 2021-09-17T10:04:22.187 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../018-ceph-common_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:22.231 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-common (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:24.539 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-base. 2021-09-17T10:04:24.555 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../019-ceph-base_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:24.609 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-base (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:25.669 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-cryptography. 2021-09-17T10:04:25.686 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../020-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2021-09-17T10:04:25.727 INFO:teuthology.orchestra.run.smithi050.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.0.0-7741-gf7edf77e-1focal [215 kB] 2021-09-17T10:04:25.728 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2021-09-17T10:04:25.731 INFO:teuthology.orchestra.run.smithi050.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-7741-gf7edf77e-1focal [20.8 kB] 2021-09-17T10:04:25.732 INFO:teuthology.orchestra.run.smithi050.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-7741-gf7edf77e-1focal [11.8 MB] 2021-09-17T10:04:26.132 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-openssl. 2021-09-17T10:04:26.151 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../021-python3-openssl_19.0.0-1build1_all.deb ... 2021-09-17T10:04:26.190 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2021-09-17T10:04:26.518 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-09-17T10:04:26.535 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../022-ceph-mgr-modules-core_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:04:26.577 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mgr-modules-core (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:26.738 INFO:teuthology.orchestra.run.smithi050.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-rgw-sts-51219/f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-7741-gf7edf77e-1focal [78.9 kB] 2021-09-17T10:04:26.982 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-bcrypt. 2021-09-17T10:04:26.999 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../023-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2021-09-17T10:04:27.048 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2021-09-17T10:04:27.338 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 177 MB in 21s (8,271 kB/s) 2021-09-17T10:04:27.384 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-cherrypy3. 2021-09-17T10:04:27.401 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../024-python3-cherrypy3_8.9.1-6_all.deb ... 2021-09-17T10:04:27.451 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2021-09-17T10:04:27.522 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-09-17T10:04:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138119 files and directories currently installed.) 2021-09-17T10:04:27.581 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2021-09-17T10:04:27.681 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2021-09-17T10:04:27.813 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-jwt. 2021-09-17T10:04:27.833 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../025-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2021-09-17T10:04:27.871 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2021-09-17T10:04:27.989 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../001-librbd1_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:28.118 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librbd1 (17.0.0-7741-gf7edf77e-1focal) over (15.2.13-0ubuntu0.20.04.2) ... 2021-09-17T10:04:28.207 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-logutils. 2021-09-17T10:04:28.223 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../026-python3-logutils_0.3.3-7_all.deb ... 2021-09-17T10:04:28.266 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-logutils (0.3.3-7) ... 2021-09-17T10:04:28.577 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-markupsafe. 2021-09-17T10:04:28.594 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../027-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2021-09-17T10:04:28.627 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2021-09-17T10:04:28.825 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../002-librados2_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:28.872 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-mako. 2021-09-17T10:04:28.889 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../028-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2021-09-17T10:04:28.936 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librados2 (17.0.0-7741-gf7edf77e-1focal) over (15.2.13-0ubuntu0.20.04.2) ... 2021-09-17T10:04:28.963 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-09-17T10:04:29.274 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-simplegeneric. 2021-09-17T10:04:29.290 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../029-python3-simplegeneric_0.8.1-3_all.deb ... 2021-09-17T10:04:29.332 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2021-09-17T10:04:29.630 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libcephfs2. 2021-09-17T10:04:29.645 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../003-libcephfs2_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:29.661 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-singledispatch. 2021-09-17T10:04:29.678 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../030-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2021-09-17T10:04:29.688 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libcephfs2 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:29.719 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2021-09-17T10:04:29.996 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-webob. 2021-09-17T10:04:30.009 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../031-python3-webob_1%3a1.8.5-2_all.deb ... 2021-09-17T10:04:30.023 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rados. 2021-09-17T10:04:30.038 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2021-09-17T10:04:30.041 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../004-python3-rados_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:30.082 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rados (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:30.358 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-soupsieve. 2021-09-17T10:04:30.376 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../032-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2021-09-17T10:04:30.416 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2021-09-17T10:04:30.441 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-09-17T10:04:30.460 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:04:30.500 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-ceph-argparse (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:30.749 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cephfs. 2021-09-17T10:04:30.752 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-bs4. 2021-09-17T10:04:30.767 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../006-python3-cephfs_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:30.769 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../033-python3-bs4_4.8.2-1_all.deb ... 2021-09-17T10:04:30.810 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cephfs (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:30.811 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2021-09-17T10:04:31.197 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-waitress. 2021-09-17T10:04:31.213 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../034-python3-waitress_1.4.1-1_all.deb ... 2021-09-17T10:04:31.259 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-waitress (1.4.1-1) ... 2021-09-17T10:04:31.274 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-ceph-common. 2021-09-17T10:04:31.291 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../007-python3-ceph-common_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:04:31.332 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-ceph-common (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:31.617 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-tempita. 2021-09-17T10:04:31.634 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../035-python3-tempita_0.5.2-6_all.deb ... 2021-09-17T10:04:31.675 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-tempita (0.5.2-6) ... 2021-09-17T10:04:31.685 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-prettytable. 2021-09-17T10:04:31.701 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2021-09-17T10:04:31.744 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2021-09-17T10:04:31.987 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rbd. 2021-09-17T10:04:31.994 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-paste. 2021-09-17T10:04:32.003 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../009-python3-rbd_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:32.013 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../036-python3-paste_3.4.0+dfsg1-1_all.deb ... 2021-09-17T10:04:32.052 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2021-09-17T10:04:32.059 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rbd (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:32.412 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2021-09-17T10:04:32.423 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-09-17T10:04:32.429 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2021-09-17T10:04:32.441 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../037-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2021-09-17T10:04:32.463 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-09-17T10:04:32.481 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2021-09-17T10:04:32.817 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pastedeploy. 2021-09-17T10:04:32.832 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-09-17T10:04:32.834 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../038-python3-pastedeploy_2.1.0-1_all.deb ... 2021-09-17T10:04:32.849 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2021-09-17T10:04:32.875 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2021-09-17T10:04:32.887 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2021-09-17T10:04:33.220 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-webtest. 2021-09-17T10:04:33.239 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../039-python3-webtest_2.0.32-2_all.deb ... 2021-09-17T10:04:33.249 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-09-17T10:04:33.265 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2021-09-17T10:04:33.278 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-webtest (2.0.32-2) ... 2021-09-17T10:04:33.308 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-09-17T10:04:33.606 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pecan. 2021-09-17T10:04:33.623 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../040-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2021-09-17T10:04:33.664 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2021-09-17T10:04:33.702 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librgw2. 2021-09-17T10:04:33.720 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../013-librgw2_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:33.761 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librgw2 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:34.053 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-werkzeug. 2021-09-17T10:04:34.070 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../041-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2021-09-17T10:04:34.111 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2021-09-17T10:04:34.433 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rgw. 2021-09-17T10:04:34.451 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../014-python3-rgw_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:34.493 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rgw (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:34.494 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2021-09-17T10:04:34.506 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../042-libsqlite3-mod-ceph_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:34.547 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:34.841 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libleveldb1d:amd64. 2021-09-17T10:04:34.857 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../015-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2021-09-17T10:04:34.900 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-09-17T10:04:34.917 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mgr. 2021-09-17T10:04:34.934 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../043-ceph-mgr_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:34.975 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mgr (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:35.222 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liboath0. 2021-09-17T10:04:35.238 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../016-liboath0_2.6.1-1.3_amd64.deb ... 2021-09-17T10:04:35.281 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2021-09-17T10:04:35.470 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mon. 2021-09-17T10:04:35.489 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../044-ceph-mon_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:35.528 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mon (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:35.607 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libradosstriper1. 2021-09-17T10:04:35.622 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../017-libradosstriper1_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:35.666 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libradosstriper1 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:36.127 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-common. 2021-09-17T10:04:36.144 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../018-ceph-common_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:36.185 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-common (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:36.682 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-osd. 2021-09-17T10:04:36.698 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../045-ceph-osd_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:36.748 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-osd (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:38.810 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-base. 2021-09-17T10:04:38.828 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../019-ceph-base_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:38.880 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-base (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:39.768 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph. 2021-09-17T10:04:39.785 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../046-ceph_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:39.826 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:40.007 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cryptography. 2021-09-17T10:04:40.024 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../020-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2021-09-17T10:04:40.064 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2021-09-17T10:04:40.162 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-fuse. 2021-09-17T10:04:40.181 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../047-ceph-fuse_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:40.220 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-fuse (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:40.494 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-openssl. 2021-09-17T10:04:40.512 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../021-python3-openssl_19.0.0-1build1_all.deb ... 2021-09-17T10:04:40.552 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2021-09-17T10:04:40.674 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mds. 2021-09-17T10:04:40.690 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../048-ceph-mds_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:40.732 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mds (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:40.847 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-09-17T10:04:40.867 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../022-ceph-mgr-modules-core_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:04:40.914 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-modules-core (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:41.202 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package cephadm. 2021-09-17T10:04:41.218 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../049-cephadm_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:41.276 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking cephadm (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:41.294 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-bcrypt. 2021-09-17T10:04:41.310 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../023-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2021-09-17T10:04:41.371 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2021-09-17T10:04:41.604 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-jinja2. 2021-09-17T10:04:41.621 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../050-python3-jinja2_2.10.1-2_all.deb ... 2021-09-17T10:04:41.682 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cherrypy3. 2021-09-17T10:04:41.687 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2021-09-17T10:04:41.697 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../024-python3-cherrypy3_8.9.1-6_all.deb ... 2021-09-17T10:04:41.738 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2021-09-17T10:04:42.016 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-09-17T10:04:42.031 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../051-ceph-mgr-cephadm_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:04:42.092 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jwt. 2021-09-17T10:04:42.098 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mgr-cephadm (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:42.109 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../025-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2021-09-17T10:04:42.150 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2021-09-17T10:04:42.436 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-repoze.lru. 2021-09-17T10:04:42.453 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../052-python3-repoze.lru_0.7-2_all.deb ... 2021-09-17T10:04:42.511 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-logutils. 2021-09-17T10:04:42.519 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2021-09-17T10:04:42.528 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../026-python3-logutils_0.3.3-7_all.deb ... 2021-09-17T10:04:42.570 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-logutils (0.3.3-7) ... 2021-09-17T10:04:42.755 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libjs-underscore. 2021-09-17T10:04:42.771 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../053-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2021-09-17T10:04:42.813 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2021-09-17T10:04:42.864 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-markupsafe. 2021-09-17T10:04:42.883 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../027-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2021-09-17T10:04:42.922 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2021-09-17T10:04:43.074 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libjs-sphinxdoc. 2021-09-17T10:04:43.090 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../054-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2021-09-17T10:04:43.123 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-09-17T10:04:43.250 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-mako. 2021-09-17T10:04:43.268 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../028-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2021-09-17T10:04:43.308 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-09-17T10:04:43.460 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-routes. 2021-09-17T10:04:43.479 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../055-python3-routes_2.4.1-2_all.deb ... 2021-09-17T10:04:43.518 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-routes (2.4.1-2) ... 2021-09-17T10:04:43.595 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-simplegeneric. 2021-09-17T10:04:43.613 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../029-python3-simplegeneric_0.8.1-3_all.deb ... 2021-09-17T10:04:43.655 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2021-09-17T10:04:43.855 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-09-17T10:04:43.870 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:04:43.913 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mgr-dashboard (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:43.964 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-singledispatch. 2021-09-17T10:04:43.983 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../030-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2021-09-17T10:04:44.022 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2021-09-17T10:04:44.342 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-webob. 2021-09-17T10:04:44.358 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../031-python3-webob_1%3a1.8.5-2_all.deb ... 2021-09-17T10:04:44.400 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2021-09-17T10:04:44.408 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-decorator. 2021-09-17T10:04:44.424 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../057-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2021-09-17T10:04:44.483 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2021-09-17T10:04:44.720 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-soupsieve. 2021-09-17T10:04:44.736 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../032-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2021-09-17T10:04:44.769 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2021-09-17T10:04:44.828 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2021-09-17T10:04:44.845 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../058-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2021-09-17T10:04:44.886 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-09-17T10:04:45.031 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-bs4. 2021-09-17T10:04:45.050 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../033-python3-bs4_4.8.2-1_all.deb ... 2021-09-17T10:04:45.081 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2021-09-17T10:04:45.214 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-scipy. 2021-09-17T10:04:45.232 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../059-python3-scipy_1.3.3-3build1_amd64.deb ... 2021-09-17T10:04:45.272 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2021-09-17T10:04:45.384 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-waitress. 2021-09-17T10:04:45.403 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../034-python3-waitress_1.4.1-1_all.deb ... 2021-09-17T10:04:45.445 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-waitress (1.4.1-1) ... 2021-09-17T10:04:45.753 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-tempita. 2021-09-17T10:04:45.770 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../035-python3-tempita_0.5.2-6_all.deb ... 2021-09-17T10:04:45.803 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-tempita (0.5.2-6) ... 2021-09-17T10:04:46.106 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-paste. 2021-09-17T10:04:46.122 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../036-python3-paste_3.4.0+dfsg1-1_all.deb ... 2021-09-17T10:04:46.164 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2021-09-17T10:04:46.518 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-09-17T10:04:46.533 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../037-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2021-09-17T10:04:46.564 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-09-17T10:04:46.582 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2021-09-17T10:04:46.584 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2021-09-17T10:04:46.622 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-09-17T10:04:46.920 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pastedeploy. 2021-09-17T10:04:46.937 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../038-python3-pastedeploy_2.1.0-1_all.deb ... 2021-09-17T10:04:46.987 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2021-09-17T10:04:47.184 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-joblib. 2021-09-17T10:04:47.202 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../061-python3-joblib_0.14.0-3_all.deb ... 2021-09-17T10:04:47.243 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-joblib (0.14.0-3) ... 2021-09-17T10:04:47.292 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-webtest. 2021-09-17T10:04:47.305 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../039-python3-webtest_2.0.32-2_all.deb ... 2021-09-17T10:04:47.375 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-webtest (2.0.32-2) ... 2021-09-17T10:04:47.686 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pecan. 2021-09-17T10:04:47.702 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../040-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2021-09-17T10:04:47.746 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2021-09-17T10:04:48.056 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-sklearn. 2021-09-17T10:04:48.071 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../062-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2021-09-17T10:04:48.113 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-09-17T10:04:48.474 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-werkzeug. 2021-09-17T10:04:48.490 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../041-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2021-09-17T10:04:48.532 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2021-09-17T10:04:48.797 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-09-17T10:04:48.814 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:04:48.888 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:48.952 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2021-09-17T10:04:48.969 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../042-libsqlite3-mod-ceph_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:49.019 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:49.405 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr. 2021-09-17T10:04:49.421 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../043-ceph-mgr_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:49.463 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:49.992 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mon. 2021-09-17T10:04:50.009 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../044-ceph-mon_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:50.025 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-cachetools. 2021-09-17T10:04:50.041 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../064-python3-cachetools_4.0.0-1_all.deb ... 2021-09-17T10:04:50.049 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mon (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:50.082 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2021-09-17T10:04:50.419 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pyasn1. 2021-09-17T10:04:50.438 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../065-python3-pyasn1_0.4.2-3build1_all.deb ... 2021-09-17T10:04:50.485 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2021-09-17T10:04:50.847 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-09-17T10:04:50.864 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../066-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2021-09-17T10:04:50.905 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-09-17T10:04:51.211 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-osd. 2021-09-17T10:04:51.231 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../045-ceph-osd_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:51.250 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-rsa. 2021-09-17T10:04:51.266 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../067-python3-rsa_4.0-3_all.deb ... 2021-09-17T10:04:51.269 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-osd (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:51.308 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-rsa (4.0-3) ... 2021-09-17T10:04:51.637 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-google-auth. 2021-09-17T10:04:51.656 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../068-python3-google-auth_1.5.1-2_all.deb ... 2021-09-17T10:04:51.695 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2021-09-17T10:04:52.058 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-blinker. 2021-09-17T10:04:52.076 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../069-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2021-09-17T10:04:52.123 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-09-17T10:04:52.460 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-oauthlib. 2021-09-17T10:04:52.478 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../070-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2021-09-17T10:04:52.518 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2021-09-17T10:04:52.880 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-requests-oauthlib. 2021-09-17T10:04:52.898 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../071-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2021-09-17T10:04:52.938 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-09-17T10:04:53.275 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-websocket. 2021-09-17T10:04:53.293 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../072-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2021-09-17T10:04:53.333 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2021-09-17T10:04:53.679 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-kubernetes. 2021-09-17T10:04:53.695 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../073-python3-kubernetes_7.0.0~a1-2_all.deb ... 2021-09-17T10:04:53.736 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2021-09-17T10:04:54.283 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-09-17T10:04:54.301 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../074-ceph-mgr-k8sevents_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:04:54.341 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:54.373 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph. 2021-09-17T10:04:54.389 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../046-ceph_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:54.431 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:54.687 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libonig5:amd64. 2021-09-17T10:04:54.708 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../075-libonig5_6.9.4-1_amd64.deb ... 2021-09-17T10:04:54.745 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2021-09-17T10:04:54.792 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-fuse. 2021-09-17T10:04:54.811 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../047-ceph-fuse_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:54.850 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-fuse (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:55.074 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libjq1:amd64. 2021-09-17T10:04:55.093 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../076-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-09-17T10:04:55.123 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-09-17T10:04:55.418 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package jq. 2021-09-17T10:04:55.434 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../077-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-09-17T10:04:55.462 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mds. 2021-09-17T10:04:55.476 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2021-09-17T10:04:55.477 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../048-ceph-mds_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:55.520 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mds (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:55.754 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package socat. 2021-09-17T10:04:55.770 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../078-socat_1.7.3.3-2_amd64.deb ... 2021-09-17T10:04:55.820 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking socat (1.7.3.3-2) ... 2021-09-17T10:04:56.107 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package cephadm. 2021-09-17T10:04:56.122 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../049-cephadm_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:56.165 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking cephadm (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:56.199 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-09-17T10:04:56.217 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../079-libxslt1.1_1.1.34-4_amd64.deb ... 2021-09-17T10:04:56.249 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2021-09-17T10:04:56.493 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jinja2. 2021-09-17T10:04:56.494 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package xmlstarlet. 2021-09-17T10:04:56.510 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../080-xmlstarlet_1.6.1-2build1_amd64.deb ... 2021-09-17T10:04:56.511 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../050-python3-jinja2_2.10.1-2_all.deb ... 2021-09-17T10:04:56.551 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2021-09-17T10:04:56.577 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2021-09-17T10:04:56.896 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-09-17T10:04:56.912 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../051-ceph-mgr-cephadm_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:04:56.980 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-cephadm (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:57.043 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-test. 2021-09-17T10:04:57.059 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../081-ceph-test_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:04:57.100 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-test (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:57.315 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-repoze.lru. 2021-09-17T10:04:57.332 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../052-python3-repoze.lru_0.7-2_all.deb ... 2021-09-17T10:04:57.382 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2021-09-17T10:04:57.702 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libjs-underscore. 2021-09-17T10:04:57.718 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../053-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2021-09-17T10:04:57.760 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2021-09-17T10:04:58.104 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libjs-sphinxdoc. 2021-09-17T10:04:58.120 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../054-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2021-09-17T10:04:58.162 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-09-17T10:04:58.482 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-routes. 2021-09-17T10:04:58.499 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../055-python3-routes_2.4.1-2_all.deb ... 2021-09-17T10:04:58.540 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-routes (2.4.1-2) ... 2021-09-17T10:04:58.893 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-09-17T10:04:58.911 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:04:58.951 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-dashboard (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:04:59.447 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-decorator. 2021-09-17T10:04:59.463 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../057-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2021-09-17T10:04:59.505 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2021-09-17T10:04:59.858 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2021-09-17T10:04:59.873 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../058-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2021-09-17T10:04:59.916 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-09-17T10:05:00.248 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-scipy. 2021-09-17T10:05:00.267 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../059-python3-scipy_1.3.3-3build1_amd64.deb ... 2021-09-17T10:05:00.310 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2021-09-17T10:05:01.439 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-volume. 2021-09-17T10:05:01.455 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../082-ceph-volume_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:05:01.497 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-volume (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:01.603 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-09-17T10:05:01.620 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2021-09-17T10:05:01.661 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-09-17T10:05:01.852 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libcephfs-dev. 2021-09-17T10:05:01.868 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../083-libcephfs-dev_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:05:01.910 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libcephfs-dev (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:02.231 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-joblib. 2021-09-17T10:05:02.249 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../061-python3-joblib_0.14.0-3_all.deb ... 2021-09-17T10:05:02.289 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-joblib (0.14.0-3) ... 2021-09-17T10:05:02.330 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package nvme-cli. 2021-09-17T10:05:02.347 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../084-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2021-09-17T10:05:02.405 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2021-09-17T10:05:02.768 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-sklearn. 2021-09-17T10:05:02.785 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../062-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2021-09-17T10:05:02.826 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-09-17T10:05:02.836 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-atomicwrites. 2021-09-17T10:05:02.854 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../085-python3-atomicwrites_1.1.5-2build1_all.deb ... 2021-09-17T10:05:02.911 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2021-09-17T10:05:03.147 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-attr. 2021-09-17T10:05:03.165 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../086-python3-attr_19.3.0-2_all.deb ... 2021-09-17T10:05:03.198 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-attr (19.3.0-2) ... 2021-09-17T10:05:03.542 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-webencodings. 2021-09-17T10:05:03.543 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-09-17T10:05:03.558 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../087-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2021-09-17T10:05:03.563 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:05:03.600 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2021-09-17T10:05:03.601 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:03.944 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-html5lib. 2021-09-17T10:05:03.962 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../088-python3-html5lib_1.0.1-2_all.deb ... 2021-09-17T10:05:04.003 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2021-09-17T10:05:04.356 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-09-17T10:05:04.376 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../089-python3-lxml_4.5.0-1ubuntu0.3_amd64.deb ... 2021-09-17T10:05:04.414 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.3) ... 2021-09-17T10:05:04.738 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cachetools. 2021-09-17T10:05:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../064-python3-cachetools_4.0.0-1_all.deb ... 2021-09-17T10:05:04.796 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2021-09-17T10:05:04.918 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-packaging. 2021-09-17T10:05:04.937 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../090-python3-packaging_20.3-1_all.deb ... 2021-09-17T10:05:04.976 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-packaging (20.3-1) ... 2021-09-17T10:05:05.182 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyasn1. 2021-09-17T10:05:05.202 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../065-python3-pyasn1_0.4.2-3build1_all.deb ... 2021-09-17T10:05:05.240 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2021-09-17T10:05:05.312 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pastescript. 2021-09-17T10:05:05.332 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../091-python3-pastescript_2.0.2-4_all.deb ... 2021-09-17T10:05:05.379 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2021-09-17T10:05:05.578 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-09-17T10:05:05.594 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../066-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2021-09-17T10:05:05.636 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-09-17T10:05:05.741 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pluggy. 2021-09-17T10:05:05.758 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../092-python3-pluggy_0.13.0-2_all.deb ... 2021-09-17T10:05:05.807 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2021-09-17T10:05:05.956 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rsa. 2021-09-17T10:05:05.975 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../067-python3-rsa_4.0-3_all.deb ... 2021-09-17T10:05:06.013 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rsa (4.0-3) ... 2021-09-17T10:05:06.165 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-psutil. 2021-09-17T10:05:06.178 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../093-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2021-09-17T10:05:06.218 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2021-09-17T10:05:06.350 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-google-auth. 2021-09-17T10:05:06.367 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../068-python3-google-auth_1.5.1-2_all.deb ... 2021-09-17T10:05:06.409 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2021-09-17T10:05:06.596 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-py. 2021-09-17T10:05:06.616 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../094-python3-py_1.8.1-1_all.deb ... 2021-09-17T10:05:06.654 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-py (1.8.1-1) ... 2021-09-17T10:05:06.728 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-blinker. 2021-09-17T10:05:06.746 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../069-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2021-09-17T10:05:06.811 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-09-17T10:05:07.008 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pyinotify. 2021-09-17T10:05:07.024 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../095-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2021-09-17T10:05:07.066 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-09-17T10:05:07.164 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-oauthlib. 2021-09-17T10:05:07.181 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../070-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2021-09-17T10:05:07.231 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2021-09-17T10:05:07.402 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-wcwidth. 2021-09-17T10:05:07.419 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../096-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2021-09-17T10:05:07.460 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2021-09-17T10:05:07.676 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-requests-oauthlib. 2021-09-17T10:05:07.693 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../071-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2021-09-17T10:05:07.743 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-09-17T10:05:07.747 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pytest. 2021-09-17T10:05:07.765 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../097-python3-pytest_4.6.9-1_all.deb ... 2021-09-17T10:05:07.796 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pytest (4.6.9-1) ... 2021-09-17T10:05:08.087 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-websocket. 2021-09-17T10:05:08.104 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../072-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2021-09-17T10:05:08.145 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2021-09-17T10:05:08.158 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-simplejson. 2021-09-17T10:05:08.175 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../098-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2021-09-17T10:05:08.209 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2021-09-17T10:05:08.490 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-kubernetes. 2021-09-17T10:05:08.506 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../073-python3-kubernetes_7.0.0~a1-2_all.deb ... 2021-09-17T10:05:08.548 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2021-09-17T10:05:08.595 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package radosgw. 2021-09-17T10:05:08.611 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../099-radosgw_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:05:08.645 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking radosgw (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:09.112 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-09-17T10:05:09.129 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../074-ceph-mgr-k8sevents_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:05:09.170 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:09.506 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libonig5:amd64. 2021-09-17T10:05:09.525 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../075-libonig5_6.9.4-1_amd64.deb ... 2021-09-17T10:05:09.564 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2021-09-17T10:05:09.940 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package rbd-fuse. 2021-09-17T10:05:09.951 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libjq1:amd64. 2021-09-17T10:05:09.957 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../100-rbd-fuse_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:05:09.967 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../076-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-09-17T10:05:09.998 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking rbd-fuse (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:10.017 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-09-17T10:05:10.303 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package jq. 2021-09-17T10:05:10.320 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../077-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-09-17T10:05:10.325 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-attr (19.3.0-2) ... 2021-09-17T10:05:10.361 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2021-09-17T10:05:10.612 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-09-17T10:05:10.639 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package socat. 2021-09-17T10:05:10.656 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../078-socat_1.7.3.3-2_amd64.deb ... 2021-09-17T10:05:10.697 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking socat (1.7.3.3-2) ... 2021-09-17T10:05:10.721 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2021-09-17T10:05:10.839 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2021-09-17T10:05:11.057 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up cephadm (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:11.077 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-09-17T10:05:11.095 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../079-libxslt1.1_1.1.34-4_amd64.deb ... 2021-09-17T10:05:11.159 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2021-09-17T10:05:11.463 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package xmlstarlet. 2021-09-17T10:05:11.483 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../080-xmlstarlet_1.6.1-2build1_amd64.deb ... 2021-09-17T10:05:11.521 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2021-09-17T10:05:11.693 INFO:teuthology.orchestra.run.smithi018.stdout:Adding system user cephadm....done 2021-09-17T10:05:11.880 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-waitress (1.4.1-1) ... 2021-09-17T10:05:11.994 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-test. 2021-09-17T10:05:12.012 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../081-ceph-test_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:05:12.052 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-test (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:12.257 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-repoze.lru (0.7-2) ... 2021-09-17T10:05:12.549 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up liboath0 (2.6.1-1.3) ... 2021-09-17T10:05:12.700 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-py (1.8.1-1) ... 2021-09-17T10:05:13.060 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-joblib (0.14.0-3) ... 2021-09-17T10:05:13.414 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2021-09-17T10:05:13.722 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-cachetools (4.0.0-1) ... 2021-09-17T10:05:14.006 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2021-09-17T10:05:14.316 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-09-17T10:05:14.625 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-ceph-argparse (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:14.951 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-09-17T10:05:15.068 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2021-09-17T10:05:15.370 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2021-09-17T10:05:15.663 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2021-09-17T10:05:16.039 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2021-09-17T10:05:16.315 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2021-09-17T10:05:16.640 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-volume. 2021-09-17T10:05:16.656 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../082-ceph-volume_17.0.0-7741-gf7edf77e-1focal_all.deb ... 2021-09-17T10:05:16.658 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2021-09-17T10:05:16.706 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-volume (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:16.951 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2021-09-17T10:05:17.045 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libcephfs-dev. 2021-09-17T10:05:17.063 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../083-libcephfs-dev_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:05:17.103 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libcephfs-dev (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:17.294 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-tempita (0.5.2-6) ... 2021-09-17T10:05:17.547 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package nvme-cli. 2021-09-17T10:05:17.564 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../084-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2021-09-17T10:05:17.569 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-packaging (20.3-1) ... 2021-09-17T10:05:17.632 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2021-09-17T10:05:17.879 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2021-09-17T10:05:17.988 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2021-09-17T10:05:18.062 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-atomicwrites. 2021-09-17T10:05:18.078 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../085-python3-atomicwrites_1.1.5-2build1_all.deb ... 2021-09-17T10:05:18.120 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2021-09-17T10:05:18.315 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2021-09-17T10:05:18.373 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-attr. 2021-09-17T10:05:18.389 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../086-python3-attr_19.3.0-2_all.deb ... 2021-09-17T10:05:18.422 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-attr (19.3.0-2) ... 2021-09-17T10:05:19.142 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2021-09-17T10:05:19.267 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-webencodings. 2021-09-17T10:05:19.284 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../087-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2021-09-17T10:05:19.325 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2021-09-17T10:05:19.543 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2021-09-17T10:05:19.678 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-html5lib. 2021-09-17T10:05:19.694 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../088-python3-html5lib_1.0.1-2_all.deb ... 2021-09-17T10:05:19.736 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2021-09-17T10:05:19.902 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up socat (1.7.3.3-2) ... 2021-09-17T10:05:20.045 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-ceph-common (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:20.081 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-09-17T10:05:20.097 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../089-python3-lxml_4.5.0-1ubuntu0.3_amd64.deb ... 2021-09-17T10:05:20.172 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.3) ... 2021-09-17T10:05:20.346 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-html5lib (1.0.1-2) ... 2021-09-17T10:05:20.635 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-packaging. 2021-09-17T10:05:20.652 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../090-python3-packaging_20.3-1_all.deb ... 2021-09-17T10:05:20.726 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-packaging (20.3-1) ... 2021-09-17T10:05:20.748 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-09-17T10:05:20.907 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pluggy (0.13.0-2) ... 2021-09-17T10:05:21.004 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pastescript. 2021-09-17T10:05:21.023 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../091-python3-pastescript_2.0.2-4_all.deb ... 2021-09-17T10:05:21.062 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2021-09-17T10:05:21.233 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-09-17T10:05:21.359 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2021-09-17T10:05:21.407 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pluggy. 2021-09-17T10:05:21.424 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../092-python3-pluggy_0.13.0-2_all.deb ... 2021-09-17T10:05:21.465 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2021-09-17T10:05:21.485 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2021-09-17T10:05:21.793 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-psutil. 2021-09-17T10:05:21.811 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../093-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2021-09-17T10:05:21.812 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2021-09-17T10:05:21.851 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2021-09-17T10:05:22.096 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-logutils (0.3.3-7) ... 2021-09-17T10:05:22.163 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-py. 2021-09-17T10:05:22.182 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../094-python3-py_1.8.1-1_all.deb ... 2021-09-17T10:05:22.221 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-py (1.8.1-1) ... 2021-09-17T10:05:22.406 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2021-09-17T10:05:22.565 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2021-09-17T10:05:22.574 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyinotify. 2021-09-17T10:05:22.594 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../095-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2021-09-17T10:05:22.632 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-09-17T10:05:22.891 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-prettytable (0.7.2-5) ... 2021-09-17T10:05:22.985 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-wcwidth. 2021-09-17T10:05:23.003 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../096-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2021-09-17T10:05:23.043 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2021-09-17T10:05:23.201 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2021-09-17T10:05:23.346 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pytest. 2021-09-17T10:05:23.364 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../097-python3-pytest_4.6.9-1_all.deb ... 2021-09-17T10:05:23.438 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pytest (4.6.9-1) ... 2021-09-17T10:05:23.552 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2021-09-17T10:05:23.808 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-simplejson. 2021-09-17T10:05:23.824 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../098-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2021-09-17T10:05:23.860 INFO:teuthology.orchestra.run.smithi018.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-09-17T10:05:23.866 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2021-09-17T10:05:24.130 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2021-09-17T10:05:24.281 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-09-17T10:05:24.328 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package radosgw. 2021-09-17T10:05:24.345 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../099-radosgw_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:05:24.386 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking radosgw (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:24.616 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2021-09-17T10:05:25.009 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2021-09-17T10:05:25.143 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-09-17T10:05:25.269 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-webob (1:1.8.5-2) ... 2021-09-17T10:05:25.595 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-09-17T10:05:25.665 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package rbd-fuse. 2021-09-17T10:05:25.681 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../100-rbd-fuse_17.0.0-7741-gf7edf77e-1focal_amd64.deb ... 2021-09-17T10:05:25.723 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking rbd-fuse (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:25.880 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up librados2 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:26.006 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2021-09-17T10:05:26.016 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-attr (19.3.0-2) ... 2021-09-17T10:05:26.461 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-09-17T10:05:26.587 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2021-09-17T10:05:26.755 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2021-09-17T10:05:26.965 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up cephadm (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:27.520 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up librgw2 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:27.568 INFO:teuthology.orchestra.run.smithi050.stdout:Adding system user cephadm....done 2021-09-17T10:05:27.646 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libsqlite3-mod-ceph (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:27.738 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-waitress (1.4.1-1) ... 2021-09-17T10:05:27.781 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-09-17T10:05:27.932 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pytest (4.6.9-1) ... 2021-09-17T10:05:28.131 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-repoze.lru (0.7-2) ... 2021-09-17T10:05:28.347 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2021-09-17T10:05:28.416 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liboath0 (2.6.1-1.3) ... 2021-09-17T10:05:28.542 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-py (1.8.1-1) ... 2021-09-17T10:05:28.651 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2021-09-17T10:05:28.894 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-joblib (0.14.0-3) ... 2021-09-17T10:05:28.952 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-bs4 (4.8.2-1) ... 2021-09-17T10:05:29.287 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2021-09-17T10:05:29.304 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libcephfs2 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:29.430 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2021-09-17T10:05:29.588 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cachetools (4.0.0-1) ... 2021-09-17T10:05:29.774 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libradosstriper1 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:29.881 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2021-09-17T10:05:29.919 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-09-17T10:05:30.182 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-09-17T10:05:30.276 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-09-17T10:05:30.500 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-ceph-argparse (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:30.801 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-09-17T10:05:30.927 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2021-09-17T10:05:31.245 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2021-09-17T10:05:31.348 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up librbd1 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:31.491 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-09-17T10:05:31.546 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2021-09-17T10:05:31.617 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2021-09-17T10:05:31.768 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.3) ... 2021-09-17T10:05:31.939 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2021-09-17T10:05:32.119 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2021-09-17T10:05:32.240 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2021-09-17T10:05:32.245 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-rsa (4.0-3) ... 2021-09-17T10:05:32.558 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2021-09-17T10:05:32.571 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-webtest (2.0.32-2) ... 2021-09-17T10:05:32.889 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mgr-modules-core (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:32.893 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2021-09-17T10:05:33.040 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-fuse (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:33.244 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-tempita (0.5.2-6) ... 2021-09-17T10:05:33.290 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-09-17T10:05:33.290 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-09-17T10:05:33.545 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-packaging (20.3-1) ... 2021-09-17T10:05:33.792 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libcephfs-dev (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:33.847 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2021-09-17T10:05:33.918 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-09-17T10:05:33.972 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2021-09-17T10:05:34.247 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pastescript (2.0.2-4) ... 2021-09-17T10:05:34.257 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2021-09-17T10:05:34.573 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2021-09-17T10:05:34.686 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2021-09-17T10:05:34.967 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-rados (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:35.054 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2021-09-17T10:05:35.118 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-rbd (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:35.244 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up rbd-fuse (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:35.378 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-routes (2.4.1-2) ... 2021-09-17T10:05:35.430 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up socat (1.7.3.3-2) ... 2021-09-17T10:05:35.556 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-ceph-common (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:35.671 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-google-auth (1.5.1-2) ... 2021-09-17T10:05:35.874 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-html5lib (1.0.1-2) ... 2021-09-17T10:05:36.067 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-rgw (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:36.195 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-cephfs (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:36.251 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-09-17T10:05:36.316 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2021-09-17T10:05:36.386 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pluggy (0.13.0-2) ... 2021-09-17T10:05:36.712 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-09-17T10:05:36.838 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2021-09-17T10:05:36.964 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2021-09-17T10:05:37.287 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-common (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:37.307 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2021-09-17T10:05:37.609 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-logutils (0.3.3-7) ... 2021-09-17T10:05:37.914 INFO:teuthology.orchestra.run.smithi018.stdout:Adding group ceph....done 2021-09-17T10:05:37.918 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2021-09-17T10:05:38.045 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2021-09-17T10:05:38.307 INFO:teuthology.orchestra.run.smithi018.stdout:Adding system user ceph....done 2021-09-17T10:05:38.363 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-prettytable (0.7.2-5) ... 2021-09-17T10:05:38.516 INFO:teuthology.orchestra.run.smithi018.stdout:Setting system user ceph properties....done 2021-09-17T10:05:38.537 INFO:teuthology.orchestra.run.smithi018.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2021-09-17T10:05:38.689 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-09-17T10:05:38.699 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2021-09-17T10:05:39.016 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2021-09-17T10:05:39.024 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-09-17T10:05:39.333 INFO:teuthology.orchestra.run.smithi050.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-09-17T10:05:39.490 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-test (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:39.601 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2021-09-17T10:05:39.616 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up radosgw (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:39.769 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-09-17T10:05:39.914 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-09-17T10:05:39.914 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-09-17T10:05:40.096 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2021-09-17T10:05:40.464 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2021-09-17T10:05:40.590 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-09-17T10:05:40.643 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-base (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:40.716 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-webob (1:1.8.5-2) ... 2021-09-17T10:05:41.028 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-09-17T10:05:41.050 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-09-17T10:05:41.368 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librados2 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:41.504 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mds (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:41.528 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2021-09-17T10:05:41.732 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-09-17T10:05:41.733 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-09-17T10:05:42.197 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mgr (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:42.455 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-09-17T10:05:42.455 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-09-17T10:05:42.915 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-osd (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:43.051 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librgw2 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:43.193 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libsqlite3-mod-ceph (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:43.328 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-09-17T10:05:43.454 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pytest (4.6.9-1) ... 2021-09-17T10:05:43.495 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-09-17T10:05:43.496 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-09-17T10:05:43.848 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2021-09-17T10:05:43.967 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mgr-k8sevents (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:44.093 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:44.157 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2021-09-17T10:05:44.236 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mon (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:44.458 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-bs4 (4.8.2-1) ... 2021-09-17T10:05:44.464 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-09-17T10:05:44.464 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-09-17T10:05:44.793 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libcephfs2 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:44.913 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mgr-cephadm (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:44.919 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2021-09-17T10:05:45.065 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:45.212 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mgr-dashboard (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:45.255 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libradosstriper1 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:45.375 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-volume (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:45.381 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-09-17T10:05:45.502 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for systemd (245.4-4ubuntu3.4) ... 2021-09-17T10:05:45.724 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-09-17T10:05:45.803 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for man-db (2.9.1-1) ... 2021-09-17T10:05:46.841 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librbd1 (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:46.972 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-09-17T10:05:47.107 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2021-09-17T10:05:47.233 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.3) ... 2021-09-17T10:05:47.559 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2021-09-17T10:05:47.702 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rsa (4.0-3) ... 2021-09-17T10:05:47.995 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-webtest (2.0.32-2) ... 2021-09-17T10:05:48.280 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-modules-core (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:48.406 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-fuse (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:48.656 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-09-17T10:05:48.657 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-09-17T10:05:49.124 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libcephfs-dev (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:49.147 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2021-09-17T10:05:49.245 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-09-17T10:05:49.513 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pastescript (2.0.2-4) ... 2021-09-17T10:05:49.552 INFO:teuthology.orchestra.run.smithi018.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-09-17T10:05:49.554 DEBUG:teuthology.parallel:result is None 2021-09-17T10:05:50.139 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2021-09-17T10:05:50.500 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rados (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:50.647 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rbd (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:50.768 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up rbd-fuse (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:50.878 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-routes (2.4.1-2) ... 2021-09-17T10:05:51.154 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-google-auth (1.5.1-2) ... 2021-09-17T10:05:51.551 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rgw (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:51.662 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cephfs (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:51.801 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2021-09-17T10:05:52.761 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-common (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:53.289 INFO:teuthology.orchestra.run.smithi050.stdout:Adding group ceph....done 2021-09-17T10:05:53.657 INFO:teuthology.orchestra.run.smithi050.stdout:Adding system user ceph....done 2021-09-17T10:05:53.859 INFO:teuthology.orchestra.run.smithi050.stdout:Setting system user ceph properties....done 2021-09-17T10:05:53.883 INFO:teuthology.orchestra.run.smithi050.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2021-09-17T10:05:54.034 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-09-17T10:05:54.364 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-09-17T10:05:54.833 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-test (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:54.959 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up radosgw (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:55.249 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-09-17T10:05:55.250 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-09-17T10:05:55.936 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-base (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:56.304 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-09-17T10:05:56.764 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mds (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:56.990 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-09-17T10:05:56.990 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-09-17T10:05:57.432 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:57.668 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-09-17T10:05:57.668 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-09-17T10:05:58.142 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-osd (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:58.732 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-09-17T10:05:58.733 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-09-17T10:05:59.177 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-k8sevents (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:59.303 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:59.480 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mon (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:05:59.706 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-09-17T10:05:59.707 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-09-17T10:06:00.164 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-cephadm (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:06:00.495 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:06:00.652 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-dashboard (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:06:00.811 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-volume (17.0.0-7741-gf7edf77e-1focal) ... 2021-09-17T10:06:00.947 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for systemd (245.4-4ubuntu3.4) ... 2021-09-17T10:06:01.264 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for man-db (2.9.1-1) ... 2021-09-17T10:06:04.442 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2021-09-17T10:06:04.851 INFO:teuthology.orchestra.run.smithi050.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-09-17T10:06:04.853 DEBUG:teuthology.parallel:result is None 2021-09-17T10:06:04.854 DEBUG:teuthology.orchestra.run.smithi018:> sudo lsb_release -is 2021-09-17T10:06:04.907 INFO:teuthology.orchestra.run.smithi018.stdout:Ubuntu 2021-09-17T10:06:04.913 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-09-17T10:06:04.913 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f7edf77e75bbfd0d8d41a561025fbfc76479a1a3 2021-09-17T10:06:05.181 DEBUG:teuthology.orchestra.run.smithi018:> dpkg-query -W -f '${Version}' ceph 2021-09-17T10:06:05.205 INFO:teuthology.orchestra.run.smithi018.stdout:17.0.0-7741-gf7edf77e-1focal 2021-09-17T10:06:05.206 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7741-gf7edf77e-1focal 2021-09-17T10:06:05.207 INFO:teuthology.task.install:The correct ceph version 17.0.0-7741-gf7edf77e-1focal is installed. 2021-09-17T10:06:05.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo lsb_release -is 2021-09-17T10:06:05.260 INFO:teuthology.orchestra.run.smithi050.stdout:Ubuntu 2021-09-17T10:06:05.266 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-09-17T10:06:05.267 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=f7edf77e75bbfd0d8d41a561025fbfc76479a1a3 2021-09-17T10:06:05.563 DEBUG:teuthology.orchestra.run.smithi050:> dpkg-query -W -f '${Version}' ceph 2021-09-17T10:06:05.587 INFO:teuthology.orchestra.run.smithi050.stdout:17.0.0-7741-gf7edf77e-1focal 2021-09-17T10:06:05.587 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7741-gf7edf77e-1focal 2021-09-17T10:06:05.588 INFO:teuthology.task.install:The correct ceph version 17.0.0-7741-gf7edf77e-1focal is installed. 2021-09-17T10:06:05.588 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-09-17T10:06:05.589 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:05.589 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-09-17T10:06:05.607 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:05.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-09-17T10:06:05.622 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-09-17T10:06:05.622 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:05.623 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/daemon-helper 2021-09-17T10:06:05.637 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-09-17T10:06:05.693 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:05.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2021-09-17T10:06:05.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-09-17T10:06:05.770 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-09-17T10:06:05.771 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:05.771 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/adjust-ulimits 2021-09-17T10:06:05.788 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-09-17T10:06:05.800 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:05.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2021-09-17T10:06:05.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-09-17T10:06:05.831 INFO:teuthology.run_tasks:Running task ceph... 2021-09-17T10:06:05.893 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-09-17T10:06:05.894 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 777 /var/log/ceph 2021-09-17T10:06:05.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2021-09-17T10:06:05.912 INFO:tasks.ceph:Disabling ceph logrotate... 2021-09-17T10:06:05.912 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/logrotate.d/ceph 2021-09-17T10:06:05.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2021-09-17T10:06:05.925 INFO:tasks.ceph:Creating extra log directories... 2021-09-17T10:06:05.925 DEBUG:teuthology.orchestra.run.smithi018:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-09-17T10:06:05.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-09-17T10:06:05.993 INFO:tasks.ceph:Creating ceph cluster c1... 2021-09-17T10:06:05.993 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-09-17T10:06:05.994 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/prsrivas-2021-09-17_09:50:01-rgw-wip-rgw-sts-51219-distro-basic-smithi/6394893', 'branch': 'wip-rgw-sts-51219', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{ubuntu_latest} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6394893', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'prsrivas-2021-09-17_09:50:01-rgw-wip-rgw-sts-51219-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-rgw-sts-51219'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'workunit': {'branch': 'wip-rgw-sts-51219', 'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3'}}, 'owner': 'scheduled_prsrivas@teuthology', 'priority': 90, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-rgw-sts-51219', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3', 'targets': {'smithi018.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlJIDWGCVw0d6mjxlh5xHp6nUKmbLatg3/YS7e43l9KwOgism12PFx3Iiqtwxc+elL4poNbk7iEMx7XcCnD/QBrYN9C5W54VzlplpQHWXUE2cJhzQ7WG2gJ+oKOF4V6BTdapegDA/7kq8Jz3Rg+OfrZkTGJCU1JUFbCw1sIc8NyEPHGarNjmw9fk6mOI76kERNC2nJjycSGerLscjHddbF9pE2bh/gPMGObzEK/ejYi5bHZGIW4Lfo+12WuBY5bxXfnl1OwnR1sdUj1cQcPk4Pc0LTqRpV1NFq2A7ExZ82kPpVJqCyNWIBrdoaryV2DNxQjEAJxPKNxEDla5a48B85teYJlzGB5nsL+YLCzvJupUEGjHPuI8ekBh75Ytzf7dkZ2wKfbYkJ5jRKGZ35Obck3O6bQpchHvwtiljzmCARJEtWhYmLrR+eaLmpzFloDoXL/iUM28EUPjAWe9ivKDTJmuoi1vKE58mBzeM5XBCPThFwkUM+9JPUetPMPomvBOc=', 'smithi050.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0ax+8bAaTUfgqqI1YjYyC3aVNh1vV6bqIxfjTx/he4L4dKnMe0z4v0rJ+gciLQfjseJ1m03CvuanS70IfSVHqbDPP82Ag6I0wqa4MLxEHO/gwfag/boLgNRPJHqq1IvxuRMDH3VeL8Y8tZZdBdPqtHUEaAUTOTqLsSaMUOvGVL17wR/+j9BtMeSFsk7hkL3EBepvGLNlRlSb96Z0IEkPOodAAv0iLWKRJMEZGSFnU6Uo4jnLmSX9mSVNuUuK5Z+ixi1n9EZ9SH1vxSjYVQ1dd2fTrEWLsyr0HLsOxcVy8QxTCuyM7zc0nAX2rWsHeVap5hDIbxcSxojIaTGhtF/54vXIgDwHTR88l24no1R1t4WVPKQ7rtpd6PafLjg8A91Dy+k7Xog4iyara0wGwCCIiLC1hQEjAPsjtfA4QMA2ckyUgMMX29xXQIIkcF/Ez54Aaim/UXYUxakohc8l+B3ZJOLDfFlX10KzjorA/xE3o9RbaA2p5nBvO+m3Auy9PBW0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '4e1f59de82924f116bd6b91664f90ef632c60a3f', 'timestamp': '2021-09-17_09:50:01', 'tube': 'smithi', 'user': 'prsrivas', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-09-17T10:06:05.994 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-09-17T10:06:05.997 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-09-17T10:06:06.003 DEBUG:teuthology.orchestra.run.smithi018:> sudo install -d -m0777 -- /var/run/ceph 2021-09-17T10:06:06.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2021-09-17T10:06:06.057 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:06.057 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2021-09-17T10:06:06.071 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-09-17T10:06:06.072 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-09-17T10:06:06.079 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-09-17T10:06:06.079 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T10:06:06.080 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 6h/6d Inode: 346 Links: 1 2021-09-17T10:06:06.080 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T10:06:06.080 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2021-09-17 10:03:18.846092846 +0000 2021-09-17T10:06:06.080 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2021-09-17 10:03:18.670092848 +0000 2021-09-17T10:06:06.081 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2021-09-17 10:03:18.670092848 +0000 2021-09-17T10:06:06.081 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2021-09-17T10:06:06.090 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2021-09-17T10:06:06.090 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2021-09-17T10:06:06.090 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000300833 s, 1.7 MB/s 2021-09-17T10:06:06.095 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-09-17T10:06:06.102 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-09-17T10:06:06.102 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T10:06:06.102 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 6h/6d Inode: 350 Links: 1 2021-09-17T10:06:06.103 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T10:06:06.103 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2021-09-17 10:03:19.070092844 +0000 2021-09-17T10:06:06.103 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2021-09-17 10:03:19.066092844 +0000 2021-09-17T10:06:06.103 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2021-09-17 10:03:19.066092844 +0000 2021-09-17T10:06:06.103 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2021-09-17T10:06:06.109 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2021-09-17T10:06:06.109 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2021-09-17T10:06:06.109 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000292211 s, 1.8 MB/s 2021-09-17T10:06:06.114 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-09-17T10:06:06.164 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-09-17T10:06:06.164 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T10:06:06.165 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 6h/6d Inode: 362 Links: 1 2021-09-17T10:06:06.165 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T10:06:06.165 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2021-09-17 10:03:19.662092837 +0000 2021-09-17T10:06:06.165 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2021-09-17 10:03:19.470092839 +0000 2021-09-17T10:06:06.165 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2021-09-17 10:03:19.470092839 +0000 2021-09-17T10:06:06.166 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2021-09-17T10:06:06.174 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2021-09-17T10:06:06.174 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2021-09-17T10:06:06.175 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000287997 s, 1.8 MB/s 2021-09-17T10:06:06.179 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-09-17T10:06:06.186 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-09-17T10:06:06.187 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T10:06:06.187 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 6h/6d Inode: 365 Links: 1 2021-09-17T10:06:06.187 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T10:06:06.187 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2021-09-17 10:03:20.066092833 +0000 2021-09-17T10:06:06.188 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2021-09-17 10:03:19.874092835 +0000 2021-09-17T10:06:06.188 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2021-09-17 10:03:19.874092835 +0000 2021-09-17T10:06:06.188 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2021-09-17T10:06:06.197 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2021-09-17T10:06:06.197 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2021-09-17T10:06:06.197 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.00028795 s, 1.8 MB/s 2021-09-17T10:06:06.202 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T10:06:06.202 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-17T10:06:06.202 INFO:tasks.ceph:Generating config... 2021-09-17T10:06:06.204 INFO:tasks.ceph:[client] debug rgw = 20 2021-09-17T10:06:06.204 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-09-17T10:06:06.204 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-09-17T10:06:06.205 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-17T10:06:06.205 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-09-17T10:06:06.205 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-09-17T10:06:06.206 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-09-17T10:06:06.206 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-09-17T10:06:06.206 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2021-09-17T10:06:06.207 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-09-17T10:06:06.207 INFO:tasks.ceph:[client] setgroup = ceph 2021-09-17T10:06:06.207 INFO:tasks.ceph:[client] setuser = ceph 2021-09-17T10:06:06.207 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-09-17T10:06:06.208 INFO:tasks.ceph:[mgr] debug ms = 1 2021-09-17T10:06:06.208 INFO:tasks.ceph:[mon] debug mon = 20 2021-09-17T10:06:06.208 INFO:tasks.ceph:[mon] debug ms = 1 2021-09-17T10:06:06.209 INFO:tasks.ceph:[mon] debug paxos = 20 2021-09-17T10:06:06.209 INFO:tasks.ceph:[osd] debug ms = 1 2021-09-17T10:06:06.209 INFO:tasks.ceph:[osd] debug osd = 20 2021-09-17T10:06:06.210 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-09-17T10:06:06.210 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-09-17T10:06:06.210 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-09-17T10:06:06.211 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-09-17T10:06:06.211 INFO:tasks.ceph:Setting up c1.mon.a... 2021-09-17T10:06:06.212 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-09-17T10:06:06.269 INFO:teuthology.orchestra.run.smithi018.stdout:creating /etc/ceph/c1.keyring 2021-09-17T10:06:06.272 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-09-17T10:06:06.310 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-17T10:06:06.329 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.18')] 2021-09-17T10:06:06.329 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.18'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-09-17T10:06:06.331 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:06.331 DEBUG:teuthology.orchestra.run.smithi018:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-09-17T10:06:06.338 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.18 --print /home/ubuntu/cephtest/c1.monmap 2021-09-17T10:06:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:06:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:06:06.419 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-09-17T10:06:06.420 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: generated fsid 7d785ba7-f183-4988-ad78-64cf45934332 2021-09-17T10:06:06.420 INFO:teuthology.orchestra.run.smithi018.stdout:setting min_mon_release = octopus 2021-09-17T10:06:06.420 INFO:teuthology.orchestra.run.smithi018.stdout:epoch 0 2021-09-17T10:06:06.420 INFO:teuthology.orchestra.run.smithi018.stdout:fsid 7d785ba7-f183-4988-ad78-64cf45934332 2021-09-17T10:06:06.420 INFO:teuthology.orchestra.run.smithi018.stdout:last_changed 2021-09-17T10:06:06.438311+0000 2021-09-17T10:06:06.421 INFO:teuthology.orchestra.run.smithi018.stdout:created 2021-09-17T10:06:06.438311+0000 2021-09-17T10:06:06.421 INFO:teuthology.orchestra.run.smithi018.stdout:min_mon_release 15 (octopus) 2021-09-17T10:06:06.421 INFO:teuthology.orchestra.run.smithi018.stdout:election_strategy: 1 2021-09-17T10:06:06.421 INFO:teuthology.orchestra.run.smithi018.stdout:0: [v2:172.21.15.18:3300/0,v1:172.21.15.18:6789/0] mon.a 2021-09-17T10:06:06.422 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-09-17T10:06:06.423 DEBUG:teuthology.orchestra.run.smithi018:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-09-17T10:06:06.472 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 7d785ba7-f183-4988-ad78-64cf45934332... 2021-09-17T10:06:06.474 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-09-17T10:06:06.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-09-17T10:06:06.509 INFO:teuthology.orchestra.run.smithi018.stdout:[global] 2021-09-17T10:06:06.509 INFO:teuthology.orchestra.run.smithi018.stdout: chdir = "" 2021-09-17T10:06:06.509 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-17T10:06:06.510 INFO:teuthology.orchestra.run.smithi018.stdout: auth supported = cephx 2021-09-17T10:06:06.510 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.510 INFO:teuthology.orchestra.run.smithi018.stdout: filestore xattr use omap = true 2021-09-17T10:06:06.510 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.511 INFO:teuthology.orchestra.run.smithi018.stdout: mon clock drift allowed = 1.000 2021-09-17T10:06:06.511 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.511 INFO:teuthology.orchestra.run.smithi018.stdout: osd crush chooseleaf type = 0 2021-09-17T10:06:06.511 INFO:teuthology.orchestra.run.smithi018.stdout: auth debug = true 2021-09-17T10:06:06.511 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.512 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on old message = true 2021-09-17T10:06:06.512 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on bug = true 2021-09-17T10:06:06.512 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.512 INFO:teuthology.orchestra.run.smithi018.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-17T10:06:06.513 INFO:teuthology.orchestra.run.smithi018.stdout: mon pg warn max object skew = 0 2021-09-17T10:06:06.513 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.513 INFO:teuthology.orchestra.run.smithi018.stdout: # disable pg_autoscaler by default for new pools 2021-09-17T10:06:06.513 INFO:teuthology.orchestra.run.smithi018.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-17T10:06:06.513 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.514 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default size = 2 2021-09-17T10:06:06.514 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.514 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow primary affinity = true 2021-09-17T10:06:06.514 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow pg remap = true 2021-09-17T10:06:06.515 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on legacy crush tunables = false 2021-09-17T10:06:06.515 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on crush straw calc version zero = false 2021-09-17T10:06:06.515 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on no sortbitwise = false 2021-09-17T10:06:06.515 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on osd down out interval zero = false 2021-09-17T10:06:06.515 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on too few osds = false 2021-09-17T10:06:06.516 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-17T10:06:06.516 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-17T10:06:06.516 INFO:teuthology.orchestra.run.smithi018.stdout: mon_allow_pool_size_one = true 2021-09-17T10:06:06.516 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.517 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-17T10:06:06.517 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.517 INFO:teuthology.orchestra.run.smithi018.stdout: osd default data pool replay window = 5 2021-09-17T10:06:06.517 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.517 INFO:teuthology.orchestra.run.smithi018.stdout: mon allow pool delete = true 2021-09-17T10:06:06.518 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.518 INFO:teuthology.orchestra.run.smithi018.stdout: mon cluster log file level = debug 2021-09-17T10:06:06.518 INFO:teuthology.orchestra.run.smithi018.stdout: debug asserts on shutdown = true 2021-09-17T10:06:06.518 INFO:teuthology.orchestra.run.smithi018.stdout: mon health detail to clog = false 2021-09-17T10:06:06.519 INFO:teuthology.orchestra.run.smithi018.stdout: mon host = 172.21.15.18 2021-09-17T10:06:06.519 INFO:teuthology.orchestra.run.smithi018.stdout: fsid = 7d785ba7-f183-4988-ad78-64cf45934332 2021-09-17T10:06:06.519 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.520 INFO:teuthology.orchestra.run.smithi018.stdout:[osd] 2021-09-17T10:06:06.520 INFO:teuthology.orchestra.run.smithi018.stdout: osd journal size = 100 2021-09-17T10:06:06.520 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.520 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub load threshold = 5.0 2021-09-17T10:06:06.520 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub max interval = 600 2021-09-17T10:06:06.521 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.521 INFO:teuthology.orchestra.run.smithi018.stdout: osd recover clone overlap = true 2021-09-17T10:06:06.521 INFO:teuthology.orchestra.run.smithi018.stdout: osd recovery max chunk = 1048576 2021-09-17T10:06:06.521 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.522 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug shutdown = true 2021-09-17T10:06:06.522 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug op order = true 2021-09-17T10:06:06.522 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify stray on activate = true 2021-09-17T10:06:06.522 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.522 INFO:teuthology.orchestra.run.smithi018.stdout: osd open classes on start = true 2021-09-17T10:06:06.523 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug pg log writeout = true 2021-09-17T10:06:06.523 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.523 INFO:teuthology.orchestra.run.smithi018.stdout: osd deep scrub update digest min age = 30 2021-09-17T10:06:06.523 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.524 INFO:teuthology.orchestra.run.smithi018.stdout: osd map max advance = 10 2021-09-17T10:06:06.524 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.524 INFO:teuthology.orchestra.run.smithi018.stdout: journal zero on create = true 2021-09-17T10:06:06.524 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.524 INFO:teuthology.orchestra.run.smithi018.stdout: filestore ondisk finisher threads = 3 2021-09-17T10:06:06.525 INFO:teuthology.orchestra.run.smithi018.stdout: filestore apply finisher threads = 3 2021-09-17T10:06:06.525 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.525 INFO:teuthology.orchestra.run.smithi018.stdout: bdev debug aio = true 2021-09-17T10:06:06.525 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug misdirected ops = true 2021-09-17T10:06:06.526 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2021-09-17T10:06:06.526 INFO:teuthology.orchestra.run.smithi018.stdout: debug osd = 20 2021-09-17T10:06:06.526 INFO:teuthology.orchestra.run.smithi018.stdout: rocksdb delete range threshold = 0 2021-09-17T10:06:06.526 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.527 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr] 2021-09-17T10:06:06.527 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2021-09-17T10:06:06.527 INFO:teuthology.orchestra.run.smithi018.stdout: debug mgr = 20 2021-09-17T10:06:06.527 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2021-09-17T10:06:06.527 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2021-09-17T10:06:06.528 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2021-09-17T10:06:06.528 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2021-09-17T10:06:06.528 INFO:teuthology.orchestra.run.smithi018.stdout: mgr/telemetry/nag = false 2021-09-17T10:06:06.528 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.528 INFO:teuthology.orchestra.run.smithi018.stdout:[mon] 2021-09-17T10:06:06.529 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2021-09-17T10:06:06.529 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2021-09-17T10:06:06.529 INFO:teuthology.orchestra.run.smithi018.stdout: debug paxos = 20 2021-09-17T10:06:06.529 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2021-09-17T10:06:06.530 INFO:teuthology.orchestra.run.smithi018.stdout: mon data avail warn = 5 2021-09-17T10:06:06.530 INFO:teuthology.orchestra.run.smithi018.stdout: mon mgr mkfs grace = 240 2021-09-17T10:06:06.530 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2021-09-17T10:06:06.530 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd reporter subtree level = osd 2021-09-17T10:06:06.530 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd prime pg temp = true 2021-09-17T10:06:06.531 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2021-09-17T10:06:06.531 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.531 INFO:teuthology.orchestra.run.smithi018.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-17T10:06:06.531 INFO:teuthology.orchestra.run.smithi018.stdout: auth mon ticket ttl = 660 # 11m 2021-09-17T10:06:06.532 INFO:teuthology.orchestra.run.smithi018.stdout: auth service ticket ttl = 240 # 4m 2021-09-17T10:06:06.532 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.532 INFO:teuthology.orchestra.run.smithi018.stdout: # don't complain about insecure global_id in the test suite 2021-09-17T10:06:06.532 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-17T10:06:06.532 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-17T10:06:06.533 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:06.533 INFO:teuthology.orchestra.run.smithi018.stdout:[client] 2021-09-17T10:06:06.533 INFO:teuthology.orchestra.run.smithi018.stdout: rgw cache enabled = true 2021-09-17T10:06:06.533 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable ops log = true 2021-09-17T10:06:06.534 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable usage log = true 2021-09-17T10:06:06.534 INFO:teuthology.orchestra.run.smithi018.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-17T10:06:06.534 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-17T10:06:06.534 INFO:teuthology.orchestra.run.smithi018.stdout: debug rgw = 20 2021-09-17T10:06:06.534 INFO:teuthology.orchestra.run.smithi018.stdout: rgw crypt require ssl = False 2021-09-17T10:06:06.535 INFO:teuthology.orchestra.run.smithi018.stdout: rgw crypt s3 kms backend = testing 2021-09-17T10:06:06.535 INFO:teuthology.orchestra.run.smithi018.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-17T10:06:06.535 INFO:teuthology.orchestra.run.smithi018.stdout: rgw curl low speed time = 300 2021-09-17T10:06:06.535 INFO:teuthology.orchestra.run.smithi018.stdout: rgw data log num shards = 4 2021-09-17T10:06:06.536 INFO:teuthology.orchestra.run.smithi018.stdout: rgw md log max shards = 4 2021-09-17T10:06:06.536 INFO:teuthology.orchestra.run.smithi018.stdout: rgw sync log trim interval = 0 2021-09-17T10:06:06.536 INFO:teuthology.orchestra.run.smithi018.stdout: rgw sync meta inject err probability = 0.1 2021-09-17T10:06:06.536 INFO:teuthology.orchestra.run.smithi018.stdout: rgw sync obj etag verify = True 2021-09-17T10:06:06.536 INFO:teuthology.orchestra.run.smithi018.stdout: setgroup = ceph 2021-09-17T10:06:06.537 INFO:teuthology.orchestra.run.smithi018.stdout: setuser = ceph 2021-09-17T10:06:06.537 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.a] 2021-09-17T10:06:06.537 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.0] 2021-09-17T10:06:06.537 INFO:teuthology.orchestra.run.smithi018.stdout: osd_max_omap_entries_per_request = 10 2021-09-17T10:06:06.538 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.1] 2021-09-17T10:06:06.538 INFO:teuthology.orchestra.run.smithi018.stdout: osd_max_omap_entries_per_request = 1000 2021-09-17T10:06:06.538 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.2] 2021-09-17T10:06:06.538 INFO:teuthology.orchestra.run.smithi018.stdout: osd_max_omap_entries_per_request = 10000 2021-09-17T10:06:06.540 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2021-09-17T10:06:06.540 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2021-09-17T10:06:06.541 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-17T10:06:06.541 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2021-09-17T10:06:06.541 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.541 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2021-09-17T10:06:06.541 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.542 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2021-09-17T10:06:06.542 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.542 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2021-09-17T10:06:06.542 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2021-09-17T10:06:06.543 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.543 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2021-09-17T10:06:06.543 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2021-09-17T10:06:06.543 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.544 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-17T10:06:06.544 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2021-09-17T10:06:06.544 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.544 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2021-09-17T10:06:06.544 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-17T10:06:06.545 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.545 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2021-09-17T10:06:06.545 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.545 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2021-09-17T10:06:06.546 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2021-09-17T10:06:06.546 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2021-09-17T10:06:06.546 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2021-09-17T10:06:06.546 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2021-09-17T10:06:06.547 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2021-09-17T10:06:06.547 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2021-09-17T10:06:06.547 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-17T10:06:06.547 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-17T10:06:06.548 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2021-09-17T10:06:06.548 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.548 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-17T10:06:06.548 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.548 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2021-09-17T10:06:06.549 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.549 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2021-09-17T10:06:06.549 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.549 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2021-09-17T10:06:06.550 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2021-09-17T10:06:06.550 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2021-09-17T10:06:06.550 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = 172.21.15.18 2021-09-17T10:06:06.550 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 7d785ba7-f183-4988-ad78-64cf45934332 2021-09-17T10:06:06.551 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.551 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2021-09-17T10:06:06.551 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2021-09-17T10:06:06.551 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.551 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2021-09-17T10:06:06.552 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2021-09-17T10:06:06.552 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.552 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2021-09-17T10:06:06.552 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2021-09-17T10:06:06.553 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.553 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2021-09-17T10:06:06.553 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2021-09-17T10:06:06.553 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2021-09-17T10:06:06.554 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.554 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2021-09-17T10:06:06.554 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2021-09-17T10:06:06.554 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.554 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2021-09-17T10:06:06.555 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.555 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2021-09-17T10:06:06.555 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.555 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2021-09-17T10:06:06.556 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.556 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2021-09-17T10:06:06.556 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2021-09-17T10:06:06.556 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.557 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2021-09-17T10:06:06.557 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2021-09-17T10:06:06.557 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2021-09-17T10:06:06.557 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2021-09-17T10:06:06.558 INFO:teuthology.orchestra.run.smithi050.stdout: rocksdb delete range threshold = 0 2021-09-17T10:06:06.558 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.558 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2021-09-17T10:06:06.558 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2021-09-17T10:06:06.559 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2021-09-17T10:06:06.559 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2021-09-17T10:06:06.559 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2021-09-17T10:06:06.559 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2021-09-17T10:06:06.560 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2021-09-17T10:06:06.560 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2021-09-17T10:06:06.560 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.560 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2021-09-17T10:06:06.561 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2021-09-17T10:06:06.561 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2021-09-17T10:06:06.561 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2021-09-17T10:06:06.561 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2021-09-17T10:06:06.561 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2021-09-17T10:06:06.562 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2021-09-17T10:06:06.562 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2021-09-17T10:06:06.562 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2021-09-17T10:06:06.562 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2021-09-17T10:06:06.562 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2021-09-17T10:06:06.563 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.563 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-17T10:06:06.563 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2021-09-17T10:06:06.563 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2021-09-17T10:06:06.564 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.564 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2021-09-17T10:06:06.564 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-17T10:06:06.564 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-17T10:06:06.564 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:06.565 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2021-09-17T10:06:06.565 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2021-09-17T10:06:06.565 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2021-09-17T10:06:06.565 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2021-09-17T10:06:06.565 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-17T10:06:06.566 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-17T10:06:06.566 INFO:teuthology.orchestra.run.smithi050.stdout: debug rgw = 20 2021-09-17T10:06:06.566 INFO:teuthology.orchestra.run.smithi050.stdout: rgw crypt require ssl = False 2021-09-17T10:06:06.566 INFO:teuthology.orchestra.run.smithi050.stdout: rgw crypt s3 kms backend = testing 2021-09-17T10:06:06.567 INFO:teuthology.orchestra.run.smithi050.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-17T10:06:06.567 INFO:teuthology.orchestra.run.smithi050.stdout: rgw curl low speed time = 300 2021-09-17T10:06:06.567 INFO:teuthology.orchestra.run.smithi050.stdout: rgw data log num shards = 4 2021-09-17T10:06:06.567 INFO:teuthology.orchestra.run.smithi050.stdout: rgw md log max shards = 4 2021-09-17T10:06:06.567 INFO:teuthology.orchestra.run.smithi050.stdout: rgw sync log trim interval = 0 2021-09-17T10:06:06.568 INFO:teuthology.orchestra.run.smithi050.stdout: rgw sync meta inject err probability = 0.1 2021-09-17T10:06:06.568 INFO:teuthology.orchestra.run.smithi050.stdout: rgw sync obj etag verify = True 2021-09-17T10:06:06.568 INFO:teuthology.orchestra.run.smithi050.stdout: setgroup = ceph 2021-09-17T10:06:06.568 INFO:teuthology.orchestra.run.smithi050.stdout: setuser = ceph 2021-09-17T10:06:06.569 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2021-09-17T10:06:06.569 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2021-09-17T10:06:06.569 INFO:teuthology.orchestra.run.smithi050.stdout: osd_max_omap_entries_per_request = 10 2021-09-17T10:06:06.569 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2021-09-17T10:06:06.569 INFO:teuthology.orchestra.run.smithi050.stdout: osd_max_omap_entries_per_request = 1000 2021-09-17T10:06:06.570 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2021-09-17T10:06:06.570 INFO:teuthology.orchestra.run.smithi050.stdout: osd_max_omap_entries_per_request = 10000 2021-09-17T10:06:06.570 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-09-17T10:06:06.571 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-09-17T10:06:06.631 INFO:tasks.ceph:Copying monmap to all nodes... 2021-09-17T10:06:06.631 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:06.632 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2021-09-17T10:06:06.639 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:06.639 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2021-09-17T10:06:06.646 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi018.front.sepia.ceph.com 2021-09-17T10:06:06.647 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:06.647 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/c1.keyring 2021-09-17T10:06:06.647 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-17T10:06:06.670 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:06.671 DEBUG:teuthology.orchestra.run.smithi018:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-09-17T10:06:06.677 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2021-09-17T10:06:06.678 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:06.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/c1.keyring 2021-09-17T10:06:06.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-17T10:06:06.705 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:06.705 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-09-17T10:06:06.710 INFO:tasks.ceph:Setting up mon nodes... 2021-09-17T10:06:06.710 INFO:tasks.ceph:Setting up mgr nodes... 2021-09-17T10:06:06.711 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-09-17T10:06:06.755 INFO:teuthology.orchestra.run.smithi018.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-09-17T10:06:06.758 INFO:tasks.ceph:Setting up mds nodes... 2021-09-17T10:06:06.758 INFO:tasks.ceph_client:Setting up client nodes... 2021-09-17T10:06:06.759 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-09-17T10:06:06.835 INFO:teuthology.orchestra.run.smithi018.stdout:creating /etc/ceph/c1.client.0.keyring 2021-09-17T10:06:06.848 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-09-17T10:06:06.878 INFO:teuthology.orchestra.run.smithi018.stdout:creating /etc/ceph/c1.client.1.keyring 2021-09-17T10:06:06.891 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-09-17T10:06:06.892 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-17T10:06:06.893 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-09-17T10:06:06.905 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T10:06:06.905 INFO:tasks.ceph:role: c1.osd.0 2021-09-17T10:06:06.905 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi018.front.sepia.ceph.com 2021-09-17T10:06:06.906 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-09-17T10:06:06.988 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-09-17T10:06:06.989 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-17T10:06:06.989 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-17T10:06:06.989 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 2021-09-17T10:06:06.989 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-17T10:06:06.990 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2021-09-17T10:06:06.990 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-17T10:06:06.990 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-17T10:06:06.990 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-17T10:06:06.990 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-17T10:06:06.992 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2021-09-17T10:06:06.992 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-09-17T10:06:07.012 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-09-17T10:06:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T10:06:07.021 INFO:teuthology.orchestra.run.smithi018.stderr:sudo: /sbin/restorecon: command not found 2021-09-17T10:06:07.021 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-09-17T10:06:07.081 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T10:06:07.082 INFO:tasks.ceph:role: c1.osd.1 2021-09-17T10:06:07.082 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi018.front.sepia.ceph.com 2021-09-17T10:06:07.083 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-09-17T10:06:07.126 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-09-17T10:06:07.127 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-17T10:06:07.127 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-17T10:06:07.127 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 2021-09-17T10:06:07.127 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-17T10:06:07.128 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2021-09-17T10:06:07.128 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-17T10:06:07.128 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-17T10:06:07.128 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-17T10:06:07.128 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-17T10:06:07.130 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2021-09-17T10:06:07.130 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-09-17T10:06:07.152 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-09-17T10:06:07.205 INFO:teuthology.orchestra.run.smithi018.stderr:sudo: /sbin/restorecon: command not found 2021-09-17T10:06:07.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T10:06:07.207 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-09-17T10:06:07.218 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T10:06:07.219 INFO:tasks.ceph:role: c1.osd.2 2021-09-17T10:06:07.219 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi018.front.sepia.ceph.com 2021-09-17T10:06:07.219 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-09-17T10:06:07.262 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-09-17T10:06:07.263 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-17T10:06:07.263 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-17T10:06:07.263 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 2021-09-17T10:06:07.264 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-17T10:06:07.264 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2021-09-17T10:06:07.264 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-17T10:06:07.264 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-17T10:06:07.264 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-17T10:06:07.265 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-17T10:06:07.266 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2021-09-17T10:06:07.266 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-09-17T10:06:07.284 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-09-17T10:06:07.337 INFO:teuthology.orchestra.run.smithi018.stderr:sudo: /sbin/restorecon: command not found 2021-09-17T10:06:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T10:06:07.339 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-17T10:06:07.417 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:07.430+0000 7f8b27c54f40 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-09-17T10:06:07.417 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:07.430+0000 7f8b27c54f40 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-09-17T10:06:07.418 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:07.430+0000 7f8b27c54f40 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-17T10:06:07.418 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:07.430+0000 7f8b27c54f40 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-17T10:06:07.418 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:07.430+0000 7f8b27c54f40 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-17T10:06:07.418 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:07.430+0000 7f8b27c54f40 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-09-17T10:06:09.634 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-09-17T10:06:09.650 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-17T10:06:09.685 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:09.698+0000 7fa9f4f09f40 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-09-17T10:06:09.685 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:09.698+0000 7fa9f4f09f40 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-09-17T10:06:09.685 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:09.698+0000 7fa9f4f09f40 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-17T10:06:09.686 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:09.698+0000 7fa9f4f09f40 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-17T10:06:09.686 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:09.698+0000 7fa9f4f09f40 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-17T10:06:09.686 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:09.698+0000 7fa9f4f09f40 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-09-17T10:06:11.885 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-09-17T10:06:11.945 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-17T10:06:11.981 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:11.994+0000 7f6873c54f40 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-09-17T10:06:11.982 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:11.998+0000 7f6873c54f40 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-09-17T10:06:11.982 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:11.998+0000 7f6873c54f40 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-17T10:06:11.982 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:11.998+0000 7f6873c54f40 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-17T10:06:11.983 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:11.998+0000 7f6873c54f40 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-17T10:06:11.983 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:06:11.998+0000 7f6873c54f40 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-09-17T10:06:14.193 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-09-17T10:06:14.210 INFO:tasks.ceph:Reading keys from all nodes... 2021-09-17T10:06:14.210 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:14.210 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2021-09-17T10:06:14.268 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:14.269 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2021-09-17T10:06:14.278 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:14.279 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2021-09-17T10:06:14.295 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:14.295 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2021-09-17T10:06:14.352 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:14.353 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2021-09-17T10:06:14.358 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:14.359 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2021-09-17T10:06:14.366 INFO:tasks.ceph:Adding keys to all mons... 2021-09-17T10:06:14.366 DEBUG:teuthology.orchestra.run.smithi018:> sudo tee -a /etc/ceph/c1.keyring 2021-09-17T10:06:14.423 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr.x] 2021-09-17T10:06:14.424 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCOaERhXtQyLhAAH5BiEnCUINHJiVrKMFDvMA== 2021-09-17T10:06:14.424 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.0] 2021-09-17T10:06:14.424 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCPaERhyAr3GRAAPy1u453rHgm88u17cwb8gQ== 2021-09-17T10:06:14.424 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.1] 2021-09-17T10:06:14.425 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCRaERhe6X1KRAAfhpHWNeg2WXGBZU5hd3wyw== 2021-09-17T10:06:14.425 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.2] 2021-09-17T10:06:14.425 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCUaERhu9AOABAAdxzjSf2xfYUVup1E337qjA== 2021-09-17T10:06:14.425 INFO:teuthology.orchestra.run.smithi018.stdout:[client.0] 2021-09-17T10:06:14.425 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCOaERhCb/7MhAAskWoUD9rzQy3GvIJLvFFxA== 2021-09-17T10:06:14.426 INFO:teuthology.orchestra.run.smithi018.stdout:[client.1] 2021-09-17T10:06:14.426 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCOaERhA2WNNRAAb/lh4ESmbBt8NiY2d3yxOw== 2021-09-17T10:06:14.427 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-09-17T10:06:14.471 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-17T10:06:14.513 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-17T10:06:14.603 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-17T10:06:14.646 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-17T10:06:14.687 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-17T10:06:14.732 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-09-17T10:06:14.732 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-09-17T10:06:14.745 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-09-17T10:06:14.850 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-09-17T10:06:14.863 DEBUG:teuthology.orchestra.run.smithi018:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-09-17T10:06:14.869 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-09-17T10:06:14.870 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-09-17T10:06:14.870 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-09-17T10:06:14.873 INFO:tasks.ceph.c1.mon.a:Started 2021-09-17T10:06:14.873 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-09-17T10:06:14.874 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-09-17T10:06:14.874 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-09-17T10:06:14.877 INFO:tasks.ceph.c1.mgr.x:Started 2021-09-17T10:06:14.877 INFO:tasks.ceph:Setting crush tunables to default 2021-09-17T10:06:14.877 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster c1 osd crush tunables default 2021-09-17T10:06:15.122 INFO:teuthology.orchestra.run.smithi018.stderr:adjusted tunables profile to default 2021-09-17T10:06:15.132 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-09-17T10:06:15.132 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:15.132 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2021-09-17T10:06:15.150 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:15.150 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2021-09-17T10:06:15.159 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:15.160 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2021-09-17T10:06:15.170 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster c1 osd new 898920cc-8136-4b98-bedd-5db8786c1af1 0 2021-09-17T10:06:15.367 INFO:teuthology.orchestra.run.smithi018.stdout:0 2021-09-17T10:06:15.379 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster c1 osd new 6d4ea972-7b6d-4310-a261-f164a70ed4cb 1 2021-09-17T10:06:15.570 INFO:teuthology.orchestra.run.smithi018.stdout:1 2021-09-17T10:06:15.584 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster c1 osd new 6a3eb6e6-a67f-4a18-89a7-142706ece696 2 2021-09-17T10:06:15.776 INFO:teuthology.orchestra.run.smithi018.stdout:2 2021-09-17T10:06:15.788 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-09-17T10:06:15.789 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-09-17T10:06:15.791 INFO:tasks.ceph.c1.osd.0:Started 2021-09-17T10:06:15.792 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-09-17T10:06:15.792 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-09-17T10:06:15.795 INFO:tasks.ceph.c1.osd.1:Started 2021-09-17T10:06:15.795 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-09-17T10:06:15.796 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-09-17T10:06:15.797 INFO:tasks.ceph.c1.osd.2:Started 2021-09-17T10:06:15.798 INFO:tasks.ceph:Waiting for OSDs to come up 2021-09-17T10:06:16.101 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-09-17T10:06:16.268 INFO:teuthology.misc.health.smithi018.stdout: 2021-09-17T10:06:16.268 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":5,"fsid":"7d785ba7-f183-4988-ad78-64cf45934332","created":"2021-09-17T10:06:15.027494+0000","modified":"2021-09-17T10:06:15.792587+0000","last_up_change":"0.000000","last_in_change":"2021-09-17T10:06:15.792587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"898920cc-8136-4b98-bedd-5db8786c1af1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d4ea972-7b6d-4310-a261-f164a70ed4cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6a3eb6e6-a67f-4a18-89a7-142706ece696","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T10:06:16.278 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-09-17T10:06:16.387 INFO:tasks.ceph.c1.osd.0.smithi018.stderr:2021-09-17T10:06:16.402+0000 7f524f88cf40 -1 Falling back to public interface 2021-09-17T10:06:16.411 INFO:tasks.ceph.c1.osd.1.smithi018.stderr:2021-09-17T10:06:16.426+0000 7f28a2e18f40 -1 Falling back to public interface 2021-09-17T10:06:16.411 INFO:tasks.ceph.c1.osd.2.smithi018.stderr:2021-09-17T10:06:16.426+0000 7fdda2a20f40 -1 Falling back to public interface 2021-09-17T10:06:19.215 INFO:tasks.ceph.c1.osd.1.smithi018.stderr:2021-09-17T10:06:19.230+0000 7f28a2e18f40 -1 osd.1 0 log_to_monitors {default=true} 2021-09-17T10:06:19.460 INFO:tasks.ceph.c1.osd.2.smithi018.stderr:2021-09-17T10:06:19.474+0000 7fdda2a20f40 -1 osd.2 0 log_to_monitors {default=true} 2021-09-17T10:06:19.481 INFO:tasks.ceph.c1.osd.0.smithi018.stderr:2021-09-17T10:06:19.494+0000 7f524f88cf40 -1 osd.0 0 log_to_monitors {default=true} 2021-09-17T10:06:21.625 INFO:tasks.ceph.c1.osd.2.smithi018.stderr:2021-09-17T10:06:21.638+0000 7fdd9e1c2700 -1 osd.2 0 waiting for initial osdmap 2021-09-17T10:06:21.628 INFO:tasks.ceph.c1.osd.2.smithi018.stderr:2021-09-17T10:06:21.642+0000 7fdd96f65700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-17T10:06:21.648 INFO:tasks.ceph.c1.osd.0.smithi018.stderr:2021-09-17T10:06:21.662+0000 7f524b02e700 -1 osd.0 0 waiting for initial osdmap 2021-09-17T10:06:21.651 INFO:tasks.ceph.c1.osd.0.smithi018.stderr:2021-09-17T10:06:21.666+0000 7f5243dd1700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-17T10:06:21.657 INFO:tasks.ceph.c1.osd.1.smithi018.stderr:2021-09-17T10:06:21.670+0000 7f289e5ba700 -1 osd.1 0 waiting for initial osdmap 2021-09-17T10:06:21.661 INFO:tasks.ceph.c1.osd.1.smithi018.stderr:2021-09-17T10:06:21.674+0000 7f289735d700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-17T10:06:22.582 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-09-17T10:06:22.857 INFO:teuthology.misc.health.smithi018.stdout: 2021-09-17T10:06:22.857 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":8,"fsid":"7d785ba7-f183-4988-ad78-64cf45934332","created":"2021-09-17T10:06:15.027494+0000","modified":"2021-09-17T10:06:22.641010+0000","last_up_change":"2021-09-17T10:06:22.641010+0000","last_in_change":"2021-09-17T10:06:15.792587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"898920cc-8136-4b98-bedd-5db8786c1af1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":14056},{"type":"v1","addr":"172.21.15.18:6801","nonce":14056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":14056},{"type":"v1","addr":"172.21.15.18:6803","nonce":14056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":14056},{"type":"v1","addr":"172.21.15.18:6807","nonce":14056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":14056},{"type":"v1","addr":"172.21.15.18:6805","nonce":14056}]},"public_addr":"172.21.15.18:6801/14056","cluster_addr":"172.21.15.18:6803/14056","heartbeat_back_addr":"172.21.15.18:6807/14056","heartbeat_front_addr":"172.21.15.18:6805/14056","state":["exists","up"]},{"osd":1,"uuid":"6d4ea972-7b6d-4310-a261-f164a70ed4cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6808","nonce":14073},{"type":"v1","addr":"172.21.15.18:6809","nonce":14073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6810","nonce":14073},{"type":"v1","addr":"172.21.15.18:6812","nonce":14073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6819","nonce":14073},{"type":"v1","addr":"172.21.15.18:6821","nonce":14073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6814","nonce":14073},{"type":"v1","addr":"172.21.15.18:6816","nonce":14073}]},"public_addr":"172.21.15.18:6809/14073","cluster_addr":"172.21.15.18:6812/14073","heartbeat_back_addr":"172.21.15.18:6821/14073","heartbeat_front_addr":"172.21.15.18:6816/14073","state":["exists","up"]},{"osd":2,"uuid":"6a3eb6e6-a67f-4a18-89a7-142706ece696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6811","nonce":14066},{"type":"v1","addr":"172.21.15.18:6813","nonce":14066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6815","nonce":14066},{"type":"v1","addr":"172.21.15.18:6817","nonce":14066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6822","nonce":14066},{"type":"v1","addr":"172.21.15.18:6823","nonce":14066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6818","nonce":14066},{"type":"v1","addr":"172.21.15.18:6820","nonce":14066}]},"public_addr":"172.21.15.18:6813/14066","cluster_addr":"172.21.15.18:6817/14066","heartbeat_back_addr":"172.21.15.18:6823/14066","heartbeat_front_addr":"172.21.15.18:6820/14066","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T10:06:22.869 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-09-17T10:06:22.869 INFO:tasks.ceph:Creating RBD pool 2021-09-17T10:06:22.870 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster c1 osd pool create rbd 8 2021-09-17T10:06:23.638 INFO:teuthology.orchestra.run.smithi018.stderr:pool 'rbd' created 2021-09-17T10:06:23.650 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-09-17T10:06:24.645 INFO:teuthology.orchestra.run.smithi018.stderr:enabled application 'rbd' on pool 'rbd' 2021-09-17T10:06:24.657 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-09-17T10:06:24.657 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-17T10:06:24.658 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-09-17T10:06:24.964 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:24.964 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":10,"fsid":"7d785ba7-f183-4988-ad78-64cf45934332","created":"2021-09-17T10:06:15.027494+0000","modified":"2021-09-17T10:06:24.644379+0000","last_up_change":"2021-09-17T10:06:22.641010+0000","last_in_change":"2021-09-17T10:06:15.792587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-17T10:06:22.759971+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-09-17T10:06:23.201014+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"898920cc-8136-4b98-bedd-5db8786c1af1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":14056},{"type":"v1","addr":"172.21.15.18:6801","nonce":14056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":14056},{"type":"v1","addr":"172.21.15.18:6803","nonce":14056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":14056},{"type":"v1","addr":"172.21.15.18:6807","nonce":14056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":14056},{"type":"v1","addr":"172.21.15.18:6805","nonce":14056}]},"public_addr":"172.21.15.18:6801/14056","cluster_addr":"172.21.15.18:6803/14056","heartbeat_back_addr":"172.21.15.18:6807/14056","heartbeat_front_addr":"172.21.15.18:6805/14056","state":["exists","up"]},{"osd":1,"uuid":"6d4ea972-7b6d-4310-a261-f164a70ed4cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6808","nonce":14073},{"type":"v1","addr":"172.21.15.18:6809","nonce":14073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6810","nonce":14073},{"type":"v1","addr":"172.21.15.18:6812","nonce":14073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6819","nonce":14073},{"type":"v1","addr":"172.21.15.18:6821","nonce":14073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6814","nonce":14073},{"type":"v1","addr":"172.21.15.18:6816","nonce":14073}]},"public_addr":"172.21.15.18:6809/14073","cluster_addr":"172.21.15.18:6812/14073","heartbeat_back_addr":"172.21.15.18:6821/14073","heartbeat_front_addr":"172.21.15.18:6816/14073","state":["exists","up"]},{"osd":2,"uuid":"6a3eb6e6-a67f-4a18-89a7-142706ece696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6811","nonce":14066},{"type":"v1","addr":"172.21.15.18:6813","nonce":14066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6815","nonce":14066},{"type":"v1","addr":"172.21.15.18:6817","nonce":14066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6822","nonce":14066},{"type":"v1","addr":"172.21.15.18:6823","nonce":14066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6818","nonce":14066},{"type":"v1","addr":"172.21.15.18:6820","nonce":14066}]},"public_addr":"172.21.15.18:6813/14066","cluster_addr":"172.21.15.18:6817/14066","heartbeat_back_addr":"172.21.15.18:6823/14066","heartbeat_front_addr":"172.21.15.18:6820/14066","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:20.531048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:20.204593+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:20.498476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T10:06:24.978 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2021-09-17T10:06:22.759971+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-09-17T10:06:23.201014+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-09-17T10:06:24.978 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get .mgr pg_num 2021-09-17T10:06:25.241 INFO:teuthology.orchestra.run.smithi018.stdout:pg_num: 1 2021-09-17T10:06:25.260 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2021-09-17T10:06:25.532 INFO:teuthology.orchestra.run.smithi018.stdout:pg_num: 8 2021-09-17T10:06:25.546 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-09-17T10:06:25.546 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-09-17T10:06:25.546 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2021-09-17T10:06:25.849 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:25.862 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6824","nonce":13873},{"type":"v1","addr":"172.21.15.18:6825","nonce":13873}]},"active_addr":"172.21.15.18:6825/13873","active_change":"2021-09-17T10:06:18.686846+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.18:0","nonce":1575182346}]},{"addrvec":[{"type":"v2","addr":"172.21.15.18:0","nonce":774412913}]},{"addrvec":[{"type":"v2","addr":"172.21.15.18:0","nonce":3148161211}]}]}} 2021-09-17T10:06:25.864 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-09-17T10:06:25.865 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-09-17T10:06:25.865 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-17T10:06:26.130 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:26.130 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":11,"fsid":"7d785ba7-f183-4988-ad78-64cf45934332","created":"2021-09-17T10:06:15.027494+0000","modified":"2021-09-17T10:06:25.646552+0000","last_up_change":"2021-09-17T10:06:22.641010+0000","last_in_change":"2021-09-17T10:06:15.792587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-17T10:06:22.759971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-09-17T10:06:23.201014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"898920cc-8136-4b98-bedd-5db8786c1af1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":14056},{"type":"v1","addr":"172.21.15.18:6801","nonce":14056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":14056},{"type":"v1","addr":"172.21.15.18:6803","nonce":14056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":14056},{"type":"v1","addr":"172.21.15.18:6807","nonce":14056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":14056},{"type":"v1","addr":"172.21.15.18:6805","nonce":14056}]},"public_addr":"172.21.15.18:6801/14056","cluster_addr":"172.21.15.18:6803/14056","heartbeat_back_addr":"172.21.15.18:6807/14056","heartbeat_front_addr":"172.21.15.18:6805/14056","state":["exists","up"]},{"osd":1,"uuid":"6d4ea972-7b6d-4310-a261-f164a70ed4cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6808","nonce":14073},{"type":"v1","addr":"172.21.15.18:6809","nonce":14073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6810","nonce":14073},{"type":"v1","addr":"172.21.15.18:6812","nonce":14073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6819","nonce":14073},{"type":"v1","addr":"172.21.15.18:6821","nonce":14073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6814","nonce":14073},{"type":"v1","addr":"172.21.15.18:6816","nonce":14073}]},"public_addr":"172.21.15.18:6809/14073","cluster_addr":"172.21.15.18:6812/14073","heartbeat_back_addr":"172.21.15.18:6821/14073","heartbeat_front_addr":"172.21.15.18:6816/14073","state":["exists","up"]},{"osd":2,"uuid":"6a3eb6e6-a67f-4a18-89a7-142706ece696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6811","nonce":14066},{"type":"v1","addr":"172.21.15.18:6813","nonce":14066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6815","nonce":14066},{"type":"v1","addr":"172.21.15.18:6817","nonce":14066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6822","nonce":14066},{"type":"v1","addr":"172.21.15.18:6823","nonce":14066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6818","nonce":14066},{"type":"v1","addr":"172.21.15.18:6820","nonce":14066}]},"public_addr":"172.21.15.18:6813/14066","cluster_addr":"172.21.15.18:6817/14066","heartbeat_back_addr":"172.21.15.18:6823/14066","heartbeat_front_addr":"172.21.15.18:6820/14066","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:20.531048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:20.204593+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:20.498476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T10:06:26.143 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-09-17T10:06:26.144 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-17T10:06:26.417 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:26.418 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":11,"fsid":"7d785ba7-f183-4988-ad78-64cf45934332","created":"2021-09-17T10:06:15.027494+0000","modified":"2021-09-17T10:06:25.646552+0000","last_up_change":"2021-09-17T10:06:22.641010+0000","last_in_change":"2021-09-17T10:06:15.792587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-17T10:06:22.759971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-09-17T10:06:23.201014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"898920cc-8136-4b98-bedd-5db8786c1af1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":14056},{"type":"v1","addr":"172.21.15.18:6801","nonce":14056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":14056},{"type":"v1","addr":"172.21.15.18:6803","nonce":14056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":14056},{"type":"v1","addr":"172.21.15.18:6807","nonce":14056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":14056},{"type":"v1","addr":"172.21.15.18:6805","nonce":14056}]},"public_addr":"172.21.15.18:6801/14056","cluster_addr":"172.21.15.18:6803/14056","heartbeat_back_addr":"172.21.15.18:6807/14056","heartbeat_front_addr":"172.21.15.18:6805/14056","state":["exists","up"]},{"osd":1,"uuid":"6d4ea972-7b6d-4310-a261-f164a70ed4cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6808","nonce":14073},{"type":"v1","addr":"172.21.15.18:6809","nonce":14073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6810","nonce":14073},{"type":"v1","addr":"172.21.15.18:6812","nonce":14073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6819","nonce":14073},{"type":"v1","addr":"172.21.15.18:6821","nonce":14073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6814","nonce":14073},{"type":"v1","addr":"172.21.15.18:6816","nonce":14073}]},"public_addr":"172.21.15.18:6809/14073","cluster_addr":"172.21.15.18:6812/14073","heartbeat_back_addr":"172.21.15.18:6821/14073","heartbeat_front_addr":"172.21.15.18:6816/14073","state":["exists","up"]},{"osd":2,"uuid":"6a3eb6e6-a67f-4a18-89a7-142706ece696","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6811","nonce":14066},{"type":"v1","addr":"172.21.15.18:6813","nonce":14066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6815","nonce":14066},{"type":"v1","addr":"172.21.15.18:6817","nonce":14066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6822","nonce":14066},{"type":"v1","addr":"172.21.15.18:6823","nonce":14066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6818","nonce":14066},{"type":"v1","addr":"172.21.15.18:6820","nonce":14066}]},"public_addr":"172.21.15.18:6813/14066","cluster_addr":"172.21.15.18:6817/14066","heartbeat_back_addr":"172.21.15.18:6823/14066","heartbeat_front_addr":"172.21.15.18:6820/14066","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:20.531048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:20.204593+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:20.498476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T10:06:26.431 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2021-09-17T10:06:26.579 INFO:teuthology.orchestra.run.smithi018.stdout:34359738371 2021-09-17T10:06:26.580 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2021-09-17T10:06:26.729 INFO:teuthology.orchestra.run.smithi018.stdout:34359738371 2021-09-17T10:06:26.730 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2021-09-17T10:06:26.887 INFO:teuthology.orchestra.run.smithi018.stdout:34359738371 2021-09-17T10:06:26.887 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-09-17T10:06:27.172 INFO:teuthology.orchestra.run.smithi018.stdout:34359738371 2021-09-17T10:06:27.186 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2021-09-17T10:06:27.186 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-09-17T10:06:27.459 INFO:teuthology.orchestra.run.smithi018.stdout:34359738370 2021-09-17T10:06:27.473 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.1 2021-09-17T10:06:28.473 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-09-17T10:06:28.760 INFO:teuthology.orchestra.run.smithi018.stdout:34359738370 2021-09-17T10:06:28.779 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.1 2021-09-17T10:06:29.780 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-09-17T10:06:30.071 INFO:teuthology.orchestra.run.smithi018.stdout:34359738372 2021-09-17T10:06:30.085 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738372 for osd.1 2021-09-17T10:06:30.086 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2021-09-17T10:06:30.358 INFO:teuthology.orchestra.run.smithi018.stdout:34359738372 2021-09-17T10:06:30.372 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738372 for osd.2 2021-09-17T10:06:30.372 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-09-17T10:06:30.372 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-09-17T10:06:30.644 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:30.647 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2021-09-17T10:06:30.657 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2021-09-17T10:06:28.696926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":33072,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":314539728,"statfs":{"total":322122547200,"available":322088681472,"internally_reserved":0,"allocated":1466368,"data_stored":1251704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.661069+0000","last_change":"2021-09-17T10:06:24.659296+0000","last_active":"2021-09-17T10:06:24.661069+0000","last_peered":"2021-09-17T10:06:24.661069+0000","last_clean":"2021-09-17T10:06:24.661069+0000","last_became_active":"2021-09-17T10:06:24.658911+0000","last_became_peered":"2021-09-17T10:06:24.658911+0000","last_unstale":"2021-09-17T10:06:24.661069+0000","last_undegraded":"2021-09-17T10:06:24.661069+0000","last_fullsized":"2021-09-17T10:06:24.661069+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.662525+0000","last_change":"2021-09-17T10:06:24.659983+0000","last_active":"2021-09-17T10:06:24.662525+0000","last_peered":"2021-09-17T10:06:24.662525+0000","last_clean":"2021-09-17T10:06:24.662525+0000","last_became_active":"2021-09-17T10:06:24.659647+0000","last_became_peered":"2021-09-17T10:06:24.659647+0000","last_unstale":"2021-09-17T10:06:24.662525+0000","last_undegraded":"2021-09-17T10:06:24.662525+0000","last_fullsized":"2021-09-17T10:06:24.662525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.663046+0000","last_change":"2021-09-17T10:06:24.661903+0000","last_active":"2021-09-17T10:06:24.663046+0000","last_peered":"2021-09-17T10:06:24.663046+0000","last_clean":"2021-09-17T10:06:24.663046+0000","last_became_active":"2021-09-17T10:06:24.661548+0000","last_became_peered":"2021-09-17T10:06:24.661548+0000","last_unstale":"2021-09-17T10:06:24.663046+0000","last_undegraded":"2021-09-17T10:06:24.663046+0000","last_fullsized":"2021-09-17T10:06:24.663046+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.661036+0000","last_change":"2021-09-17T10:06:24.659416+0000","last_active":"2021-09-17T10:06:24.661036+0000","last_peered":"2021-09-17T10:06:24.661036+0000","last_clean":"2021-09-17T10:06:24.661036+0000","last_became_active":"2021-09-17T10:06:24.659042+0000","last_became_peered":"2021-09-17T10:06:24.659042+0000","last_unstale":"2021-09-17T10:06:24.661036+0000","last_undegraded":"2021-09-17T10:06:24.661036+0000","last_fullsized":"2021-09-17T10:06:24.661036+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.657535+0000","last_change":"2021-09-17T10:06:24.657535+0000","last_active":"2021-09-17T10:06:24.657535+0000","last_peered":"2021-09-17T10:06:24.657535+0000","last_clean":"2021-09-17T10:06:24.657535+0000","last_became_active":"2021-09-17T10:06:24.657157+0000","last_became_peered":"2021-09-17T10:06:24.657157+0000","last_unstale":"2021-09-17T10:06:24.657535+0000","last_undegraded":"2021-09-17T10:06:24.657535+0000","last_fullsized":"2021-09-17T10:06:24.657535+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.659463+0000","last_change":"2021-09-17T10:06:24.659463+0000","last_active":"2021-09-17T10:06:24.659463+0000","last_peered":"2021-09-17T10:06:24.659463+0000","last_clean":"2021-09-17T10:06:24.659463+0000","last_became_active":"2021-09-17T10:06:24.658985+0000","last_became_peered":"2021-09-17T10:06:24.658985+0000","last_unstale":"2021-09-17T10:06:24.659463+0000","last_undegraded":"2021-09-17T10:06:24.659463+0000","last_fullsized":"2021-09-17T10:06:24.659463+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.660996+0000","last_change":"2021-09-17T10:06:24.660996+0000","last_active":"2021-09-17T10:06:24.660996+0000","last_peered":"2021-09-17T10:06:24.660996+0000","last_clean":"2021-09-17T10:06:24.660996+0000","last_became_active":"2021-09-17T10:06:24.660648+0000","last_became_peered":"2021-09-17T10:06:24.660648+0000","last_unstale":"2021-09-17T10:06:24.660996+0000","last_undegraded":"2021-09-17T10:06:24.660996+0000","last_fullsized":"2021-09-17T10:06:24.660996+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.661054+0000","last_change":"2021-09-17T10:06:24.659509+0000","last_active":"2021-09-17T10:06:24.661054+0000","last_peered":"2021-09-17T10:06:24.661054+0000","last_clean":"2021-09-17T10:06:24.661054+0000","last_became_active":"2021-09-17T10:06:24.659119+0000","last_became_peered":"2021-09-17T10:06:24.659119+0000","last_unstale":"2021-09-17T10:06:24.661054+0000","last_undegraded":"2021-09-17T10:06:24.661054+0000","last_fullsized":"2021-09-17T10:06:24.661054+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:25.274741+0000","last_change":"2021-09-17T10:06:24.663643+0000","last_active":"2021-09-17T10:06:25.274741+0000","last_peered":"2021-09-17T10:06:25.274741+0000","last_clean":"2021-09-17T10:06:25.274741+0000","last_became_active":"2021-09-17T10:06:24.662567+0000","last_became_peered":"2021-09-17T10:06:24.662567+0000","last_unstale":"2021-09-17T10:06:25.274741+0000","last_undegraded":"2021-09-17T10:06:25.274741+0000","last_fullsized":"2021-09-17T10:06:25.274741+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7520,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7424,"kb_avail":104850080,"statfs":{"total":107374182400,"available":107366481920,"internally_reserved":0,"allocated":90112,"data_stored":22388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7602176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":12780,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":11968,"kb_avail":104844820,"statfs":{"total":107374182400,"available":107361095680,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12255232},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":12772,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":11968,"kb_avail":104844828,"statfs":{"total":107374182400,"available":107361103872,"internally_reserved":0,"allocated":684032,"data_stored":612756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12255232},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-17T10:06:30.658 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-09-17T10:06:30.929 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:30.932 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2021-09-17T10:06:30.943 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2021-09-17T10:06:30.697536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":33072,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":314539728,"statfs":{"total":322122547200,"available":322088681472,"internally_reserved":0,"allocated":1466368,"data_stored":1251704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000610"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.661069+0000","last_change":"2021-09-17T10:06:24.659296+0000","last_active":"2021-09-17T10:06:24.661069+0000","last_peered":"2021-09-17T10:06:24.661069+0000","last_clean":"2021-09-17T10:06:24.661069+0000","last_became_active":"2021-09-17T10:06:24.658911+0000","last_became_peered":"2021-09-17T10:06:24.658911+0000","last_unstale":"2021-09-17T10:06:24.661069+0000","last_undegraded":"2021-09-17T10:06:24.661069+0000","last_fullsized":"2021-09-17T10:06:24.661069+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.662525+0000","last_change":"2021-09-17T10:06:24.659983+0000","last_active":"2021-09-17T10:06:24.662525+0000","last_peered":"2021-09-17T10:06:24.662525+0000","last_clean":"2021-09-17T10:06:24.662525+0000","last_became_active":"2021-09-17T10:06:24.659647+0000","last_became_peered":"2021-09-17T10:06:24.659647+0000","last_unstale":"2021-09-17T10:06:24.662525+0000","last_undegraded":"2021-09-17T10:06:24.662525+0000","last_fullsized":"2021-09-17T10:06:24.662525+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.663046+0000","last_change":"2021-09-17T10:06:24.661903+0000","last_active":"2021-09-17T10:06:24.663046+0000","last_peered":"2021-09-17T10:06:24.663046+0000","last_clean":"2021-09-17T10:06:24.663046+0000","last_became_active":"2021-09-17T10:06:24.661548+0000","last_became_peered":"2021-09-17T10:06:24.661548+0000","last_unstale":"2021-09-17T10:06:24.663046+0000","last_undegraded":"2021-09-17T10:06:24.663046+0000","last_fullsized":"2021-09-17T10:06:24.663046+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.661036+0000","last_change":"2021-09-17T10:06:24.659416+0000","last_active":"2021-09-17T10:06:24.661036+0000","last_peered":"2021-09-17T10:06:24.661036+0000","last_clean":"2021-09-17T10:06:24.661036+0000","last_became_active":"2021-09-17T10:06:24.659042+0000","last_became_peered":"2021-09-17T10:06:24.659042+0000","last_unstale":"2021-09-17T10:06:24.661036+0000","last_undegraded":"2021-09-17T10:06:24.661036+0000","last_fullsized":"2021-09-17T10:06:24.661036+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.657535+0000","last_change":"2021-09-17T10:06:24.657535+0000","last_active":"2021-09-17T10:06:24.657535+0000","last_peered":"2021-09-17T10:06:24.657535+0000","last_clean":"2021-09-17T10:06:24.657535+0000","last_became_active":"2021-09-17T10:06:24.657157+0000","last_became_peered":"2021-09-17T10:06:24.657157+0000","last_unstale":"2021-09-17T10:06:24.657535+0000","last_undegraded":"2021-09-17T10:06:24.657535+0000","last_fullsized":"2021-09-17T10:06:24.657535+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.659463+0000","last_change":"2021-09-17T10:06:24.659463+0000","last_active":"2021-09-17T10:06:24.659463+0000","last_peered":"2021-09-17T10:06:24.659463+0000","last_clean":"2021-09-17T10:06:24.659463+0000","last_became_active":"2021-09-17T10:06:24.658985+0000","last_became_peered":"2021-09-17T10:06:24.658985+0000","last_unstale":"2021-09-17T10:06:24.659463+0000","last_undegraded":"2021-09-17T10:06:24.659463+0000","last_fullsized":"2021-09-17T10:06:24.659463+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.660996+0000","last_change":"2021-09-17T10:06:24.660996+0000","last_active":"2021-09-17T10:06:24.660996+0000","last_peered":"2021-09-17T10:06:24.660996+0000","last_clean":"2021-09-17T10:06:24.660996+0000","last_became_active":"2021-09-17T10:06:24.660648+0000","last_became_peered":"2021-09-17T10:06:24.660648+0000","last_unstale":"2021-09-17T10:06:24.660996+0000","last_undegraded":"2021-09-17T10:06:24.660996+0000","last_fullsized":"2021-09-17T10:06:24.660996+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:24.661054+0000","last_change":"2021-09-17T10:06:24.659509+0000","last_active":"2021-09-17T10:06:24.661054+0000","last_peered":"2021-09-17T10:06:24.661054+0000","last_clean":"2021-09-17T10:06:24.661054+0000","last_became_active":"2021-09-17T10:06:24.659119+0000","last_became_peered":"2021-09-17T10:06:24.659119+0000","last_unstale":"2021-09-17T10:06:24.661054+0000","last_undegraded":"2021-09-17T10:06:24.661054+0000","last_fullsized":"2021-09-17T10:06:24.661054+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:25.274741+0000","last_change":"2021-09-17T10:06:24.663643+0000","last_active":"2021-09-17T10:06:25.274741+0000","last_peered":"2021-09-17T10:06:25.274741+0000","last_clean":"2021-09-17T10:06:25.274741+0000","last_became_active":"2021-09-17T10:06:24.662567+0000","last_became_peered":"2021-09-17T10:06:24.662567+0000","last_unstale":"2021-09-17T10:06:25.274741+0000","last_undegraded":"2021-09-17T10:06:25.274741+0000","last_fullsized":"2021-09-17T10:06:25.274741+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:23.642594+0000","last_clean_scrub_stamp":"2021-09-17T10:06:23.642594+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7520,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7424,"kb_avail":104850080,"statfs":{"total":107374182400,"available":107366481920,"internally_reserved":0,"allocated":90112,"data_stored":22388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7602176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":12780,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":11968,"kb_avail":104844820,"statfs":{"total":107374182400,"available":107361095680,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12255232},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":12772,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":11968,"kb_avail":104844828,"statfs":{"total":107374182400,"available":107361103872,"internally_reserved":0,"allocated":684032,"data_stored":612756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12255232},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-17T10:06:30.944 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-09-17T10:06:30.944 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-09-17T10:06:30.944 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-09-17T10:06:30.945 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2021-09-17T10:06:31.269 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.269 INFO:teuthology.orchestra.run.smithi018.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-09-17T10:06:31.282 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-09-17T10:06:31.282 INFO:teuthology.run_tasks:Running task ceph... 2021-09-17T10:06:31.293 INFO:tasks.ceph:Creating ceph cluster c2... 2021-09-17T10:06:31.293 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-09-17T10:06:31.293 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/prsrivas-2021-09-17_09:50:01-rgw-wip-rgw-sts-51219-distro-basic-smithi/6394893', 'branch': 'wip-rgw-sts-51219', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{ubuntu_latest} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6394893', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'prsrivas-2021-09-17_09:50:01-rgw-wip-rgw-sts-51219-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-rgw-sts-51219'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'workunit': {'branch': 'wip-rgw-sts-51219', 'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3'}}, 'owner': 'scheduled_prsrivas@teuthology', 'priority': 90, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-rgw-sts-51219', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3', 'targets': {'smithi018.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlJIDWGCVw0d6mjxlh5xHp6nUKmbLatg3/YS7e43l9KwOgism12PFx3Iiqtwxc+elL4poNbk7iEMx7XcCnD/QBrYN9C5W54VzlplpQHWXUE2cJhzQ7WG2gJ+oKOF4V6BTdapegDA/7kq8Jz3Rg+OfrZkTGJCU1JUFbCw1sIc8NyEPHGarNjmw9fk6mOI76kERNC2nJjycSGerLscjHddbF9pE2bh/gPMGObzEK/ejYi5bHZGIW4Lfo+12WuBY5bxXfnl1OwnR1sdUj1cQcPk4Pc0LTqRpV1NFq2A7ExZ82kPpVJqCyNWIBrdoaryV2DNxQjEAJxPKNxEDla5a48B85teYJlzGB5nsL+YLCzvJupUEGjHPuI8ekBh75Ytzf7dkZ2wKfbYkJ5jRKGZ35Obck3O6bQpchHvwtiljzmCARJEtWhYmLrR+eaLmpzFloDoXL/iUM28EUPjAWe9ivKDTJmuoi1vKE58mBzeM5XBCPThFwkUM+9JPUetPMPomvBOc=', 'smithi050.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0ax+8bAaTUfgqqI1YjYyC3aVNh1vV6bqIxfjTx/he4L4dKnMe0z4v0rJ+gciLQfjseJ1m03CvuanS70IfSVHqbDPP82Ag6I0wqa4MLxEHO/gwfag/boLgNRPJHqq1IvxuRMDH3VeL8Y8tZZdBdPqtHUEaAUTOTqLsSaMUOvGVL17wR/+j9BtMeSFsk7hkL3EBepvGLNlRlSb96Z0IEkPOodAAv0iLWKRJMEZGSFnU6Uo4jnLmSX9mSVNuUuK5Z+ixi1n9EZ9SH1vxSjYVQ1dd2fTrEWLsyr0HLsOxcVy8QxTCuyM7zc0nAX2rWsHeVap5hDIbxcSxojIaTGhtF/54vXIgDwHTR88l24no1R1t4WVPKQ7rtpd6PafLjg8A91Dy+k7Xog4iyara0wGwCCIiLC1hQEjAPsjtfA4QMA2ckyUgMMX29xXQIIkcF/Ez54Aaim/UXYUxakohc8l+B3ZJOLDfFlX10KzjorA/xE3o9RbaA2p5nBvO+m3Auy9PBW0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'f7edf77e75bbfd0d8d41a561025fbfc76479a1a3', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '4e1f59de82924f116bd6b91664f90ef632c60a3f', 'timestamp': '2021-09-17_09:50:01', 'tube': 'smithi', 'user': 'prsrivas', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-09-17T10:06:31.294 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-09-17T10:06:31.297 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-09-17T10:06:31.305 DEBUG:teuthology.orchestra.run.smithi018:> sudo install -d -m0777 -- /var/run/ceph 2021-09-17T10:06:31.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2021-09-17T10:06:31.321 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:31.322 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2021-09-17T10:06:31.369 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-09-17T10:06:31.370 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-09-17T10:06:31.377 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-09-17T10:06:31.377 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T10:06:31.377 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 346 Links: 1 2021-09-17T10:06:31.378 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T10:06:31.378 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2021-09-17 10:03:20.398013414 +0000 2021-09-17T10:06:31.378 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2021-09-17 10:03:20.394013414 +0000 2021-09-17T10:06:31.378 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2021-09-17 10:03:20.394013414 +0000 2021-09-17T10:06:31.379 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2021-09-17T10:06:31.388 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2021-09-17T10:06:31.388 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2021-09-17T10:06:31.388 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000309343 s, 1.7 MB/s 2021-09-17T10:06:31.393 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-09-17T10:06:31.401 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-09-17T10:06:31.401 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T10:06:31.401 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 355 Links: 1 2021-09-17T10:06:31.401 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T10:06:31.402 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2021-09-17 10:03:20.986013407 +0000 2021-09-17T10:06:31.402 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2021-09-17 10:03:20.806013409 +0000 2021-09-17T10:06:31.402 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2021-09-17 10:03:20.806013409 +0000 2021-09-17T10:06:31.402 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2021-09-17T10:06:31.411 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2021-09-17T10:06:31.411 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2021-09-17T10:06:31.412 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.00031094 s, 1.6 MB/s 2021-09-17T10:06:31.415 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-09-17T10:06:31.465 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-09-17T10:06:31.466 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T10:06:31.466 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 361 Links: 1 2021-09-17T10:06:31.466 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T10:06:31.466 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2021-09-17 10:03:21.386013403 +0000 2021-09-17T10:06:31.467 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2021-09-17 10:03:21.210013405 +0000 2021-09-17T10:06:31.467 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2021-09-17 10:03:21.210013405 +0000 2021-09-17T10:06:31.467 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2021-09-17T10:06:31.476 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2021-09-17T10:06:31.476 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2021-09-17T10:06:31.476 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000293137 s, 1.7 MB/s 2021-09-17T10:06:31.480 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-09-17T10:06:31.488 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-09-17T10:06:31.488 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-17T10:06:31.488 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 367 Links: 1 2021-09-17T10:06:31.488 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-17T10:06:31.489 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2021-09-17 10:03:21.790013399 +0000 2021-09-17T10:06:31.489 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2021-09-17 10:03:21.610013400 +0000 2021-09-17T10:06:31.489 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2021-09-17 10:03:21.610013400 +0000 2021-09-17T10:06:31.489 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2021-09-17T10:06:31.495 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2021-09-17T10:06:31.495 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2021-09-17T10:06:31.496 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.00026554 s, 1.9 MB/s 2021-09-17T10:06:31.500 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T10:06:31.500 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-17T10:06:31.500 INFO:tasks.ceph:Generating config... 2021-09-17T10:06:31.502 INFO:tasks.ceph:[client] debug rgw = 20 2021-09-17T10:06:31.502 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-09-17T10:06:31.502 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-09-17T10:06:31.503 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-17T10:06:31.503 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-09-17T10:06:31.503 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-09-17T10:06:31.504 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-09-17T10:06:31.504 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-09-17T10:06:31.504 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2021-09-17T10:06:31.504 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-09-17T10:06:31.505 INFO:tasks.ceph:[client] setgroup = ceph 2021-09-17T10:06:31.505 INFO:tasks.ceph:[client] setuser = ceph 2021-09-17T10:06:31.505 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-09-17T10:06:31.506 INFO:tasks.ceph:[mgr] debug ms = 1 2021-09-17T10:06:31.506 INFO:tasks.ceph:[mon] debug mon = 20 2021-09-17T10:06:31.506 INFO:tasks.ceph:[mon] debug ms = 1 2021-09-17T10:06:31.507 INFO:tasks.ceph:[mon] debug paxos = 20 2021-09-17T10:06:31.507 INFO:tasks.ceph:[osd] debug ms = 1 2021-09-17T10:06:31.507 INFO:tasks.ceph:[osd] debug osd = 20 2021-09-17T10:06:31.507 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-09-17T10:06:31.508 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-09-17T10:06:31.508 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-09-17T10:06:31.508 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-09-17T10:06:31.509 INFO:tasks.ceph:Setting up c2.mon.a... 2021-09-17T10:06:31.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-09-17T10:06:31.557 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/c2.keyring 2021-09-17T10:06:31.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-09-17T10:06:31.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-17T10:06:31.609 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.50')] 2021-09-17T10:06:31.610 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.50'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-09-17T10:06:31.611 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:31.611 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-09-17T10:06:31.617 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.50 --print /home/ubuntu/cephtest/c2.monmap 2021-09-17T10:06:31.693 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:06:31.693 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:06:31.694 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-09-17T10:06:31.694 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: generated fsid 5e649385-aff9-4de1-8a2e-b2ceb667c16c 2021-09-17T10:06:31.694 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = octopus 2021-09-17T10:06:31.695 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 0 2021-09-17T10:06:31.695 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 5e649385-aff9-4de1-8a2e-b2ceb667c16c 2021-09-17T10:06:31.695 INFO:teuthology.orchestra.run.smithi050.stdout:last_changed 2021-09-17T10:06:31.715600+0000 2021-09-17T10:06:31.695 INFO:teuthology.orchestra.run.smithi050.stdout:created 2021-09-17T10:06:31.715600+0000 2021-09-17T10:06:31.695 INFO:teuthology.orchestra.run.smithi050.stdout:min_mon_release 15 (octopus) 2021-09-17T10:06:31.696 INFO:teuthology.orchestra.run.smithi050.stdout:election_strategy: 1 2021-09-17T10:06:31.696 INFO:teuthology.orchestra.run.smithi050.stdout:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2021-09-17T10:06:31.696 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-09-17T10:06:31.697 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-09-17T10:06:31.704 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 5e649385-aff9-4de1-8a2e-b2ceb667c16c... 2021-09-17T10:06:31.705 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-09-17T10:06:31.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-09-17T10:06:31.749 INFO:teuthology.orchestra.run.smithi018.stdout:[global] 2021-09-17T10:06:31.750 INFO:teuthology.orchestra.run.smithi018.stdout: chdir = "" 2021-09-17T10:06:31.750 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-17T10:06:31.750 INFO:teuthology.orchestra.run.smithi018.stdout: auth supported = cephx 2021-09-17T10:06:31.750 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.751 INFO:teuthology.orchestra.run.smithi018.stdout: filestore xattr use omap = true 2021-09-17T10:06:31.751 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.751 INFO:teuthology.orchestra.run.smithi018.stdout: mon clock drift allowed = 1.000 2021-09-17T10:06:31.751 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.751 INFO:teuthology.orchestra.run.smithi018.stdout: osd crush chooseleaf type = 0 2021-09-17T10:06:31.752 INFO:teuthology.orchestra.run.smithi018.stdout: auth debug = true 2021-09-17T10:06:31.752 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.752 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on old message = true 2021-09-17T10:06:31.752 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on bug = true 2021-09-17T10:06:31.752 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.753 INFO:teuthology.orchestra.run.smithi018.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-17T10:06:31.753 INFO:teuthology.orchestra.run.smithi018.stdout: mon pg warn max object skew = 0 2021-09-17T10:06:31.753 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.753 INFO:teuthology.orchestra.run.smithi018.stdout: # disable pg_autoscaler by default for new pools 2021-09-17T10:06:31.754 INFO:teuthology.orchestra.run.smithi018.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-17T10:06:31.754 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.754 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default size = 2 2021-09-17T10:06:31.754 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.755 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow primary affinity = true 2021-09-17T10:06:31.755 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow pg remap = true 2021-09-17T10:06:31.755 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on legacy crush tunables = false 2021-09-17T10:06:31.755 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on crush straw calc version zero = false 2021-09-17T10:06:31.755 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on no sortbitwise = false 2021-09-17T10:06:31.756 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on osd down out interval zero = false 2021-09-17T10:06:31.756 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on too few osds = false 2021-09-17T10:06:31.756 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-17T10:06:31.756 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-17T10:06:31.756 INFO:teuthology.orchestra.run.smithi018.stdout: mon_allow_pool_size_one = true 2021-09-17T10:06:31.757 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.757 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-17T10:06:31.757 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.757 INFO:teuthology.orchestra.run.smithi018.stdout: osd default data pool replay window = 5 2021-09-17T10:06:31.757 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.758 INFO:teuthology.orchestra.run.smithi018.stdout: mon allow pool delete = true 2021-09-17T10:06:31.758 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.758 INFO:teuthology.orchestra.run.smithi018.stdout: mon cluster log file level = debug 2021-09-17T10:06:31.758 INFO:teuthology.orchestra.run.smithi018.stdout: debug asserts on shutdown = true 2021-09-17T10:06:31.759 INFO:teuthology.orchestra.run.smithi018.stdout: mon health detail to clog = false 2021-09-17T10:06:31.759 INFO:teuthology.orchestra.run.smithi018.stdout: mon host = 172.21.15.50 2021-09-17T10:06:31.759 INFO:teuthology.orchestra.run.smithi018.stdout: fsid = 5e649385-aff9-4de1-8a2e-b2ceb667c16c 2021-09-17T10:06:31.759 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.759 INFO:teuthology.orchestra.run.smithi018.stdout:[osd] 2021-09-17T10:06:31.760 INFO:teuthology.orchestra.run.smithi018.stdout: osd journal size = 100 2021-09-17T10:06:31.760 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.760 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub load threshold = 5.0 2021-09-17T10:06:31.760 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub max interval = 600 2021-09-17T10:06:31.760 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.761 INFO:teuthology.orchestra.run.smithi018.stdout: osd recover clone overlap = true 2021-09-17T10:06:31.761 INFO:teuthology.orchestra.run.smithi018.stdout: osd recovery max chunk = 1048576 2021-09-17T10:06:31.761 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.761 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug shutdown = true 2021-09-17T10:06:31.762 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug op order = true 2021-09-17T10:06:31.762 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify stray on activate = true 2021-09-17T10:06:31.762 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.762 INFO:teuthology.orchestra.run.smithi018.stdout: osd open classes on start = true 2021-09-17T10:06:31.762 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug pg log writeout = true 2021-09-17T10:06:31.763 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.763 INFO:teuthology.orchestra.run.smithi018.stdout: osd deep scrub update digest min age = 30 2021-09-17T10:06:31.763 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.763 INFO:teuthology.orchestra.run.smithi018.stdout: osd map max advance = 10 2021-09-17T10:06:31.764 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.764 INFO:teuthology.orchestra.run.smithi018.stdout: journal zero on create = true 2021-09-17T10:06:31.764 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.764 INFO:teuthology.orchestra.run.smithi018.stdout: filestore ondisk finisher threads = 3 2021-09-17T10:06:31.764 INFO:teuthology.orchestra.run.smithi018.stdout: filestore apply finisher threads = 3 2021-09-17T10:06:31.765 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.765 INFO:teuthology.orchestra.run.smithi018.stdout: bdev debug aio = true 2021-09-17T10:06:31.765 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug misdirected ops = true 2021-09-17T10:06:31.765 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2021-09-17T10:06:31.766 INFO:teuthology.orchestra.run.smithi018.stdout: debug osd = 20 2021-09-17T10:06:31.766 INFO:teuthology.orchestra.run.smithi018.stdout: rocksdb delete range threshold = 0 2021-09-17T10:06:31.766 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.766 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr] 2021-09-17T10:06:31.766 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2021-09-17T10:06:31.767 INFO:teuthology.orchestra.run.smithi018.stdout: debug mgr = 20 2021-09-17T10:06:31.767 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2021-09-17T10:06:31.767 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2021-09-17T10:06:31.767 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2021-09-17T10:06:31.768 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2021-09-17T10:06:31.768 INFO:teuthology.orchestra.run.smithi018.stdout: mgr/telemetry/nag = false 2021-09-17T10:06:31.768 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.768 INFO:teuthology.orchestra.run.smithi018.stdout:[mon] 2021-09-17T10:06:31.768 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2021-09-17T10:06:31.769 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2021-09-17T10:06:31.769 INFO:teuthology.orchestra.run.smithi018.stdout: debug paxos = 20 2021-09-17T10:06:31.769 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2021-09-17T10:06:31.769 INFO:teuthology.orchestra.run.smithi018.stdout: mon data avail warn = 5 2021-09-17T10:06:31.770 INFO:teuthology.orchestra.run.smithi018.stdout: mon mgr mkfs grace = 240 2021-09-17T10:06:31.770 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2021-09-17T10:06:31.770 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd reporter subtree level = osd 2021-09-17T10:06:31.770 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd prime pg temp = true 2021-09-17T10:06:31.770 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2021-09-17T10:06:31.771 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.771 INFO:teuthology.orchestra.run.smithi018.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-17T10:06:31.771 INFO:teuthology.orchestra.run.smithi018.stdout: auth mon ticket ttl = 660 # 11m 2021-09-17T10:06:31.771 INFO:teuthology.orchestra.run.smithi018.stdout: auth service ticket ttl = 240 # 4m 2021-09-17T10:06:31.772 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.772 INFO:teuthology.orchestra.run.smithi018.stdout: # don't complain about insecure global_id in the test suite 2021-09-17T10:06:31.772 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-17T10:06:31.772 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-17T10:06:31.772 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:06:31.773 INFO:teuthology.orchestra.run.smithi018.stdout:[client] 2021-09-17T10:06:31.773 INFO:teuthology.orchestra.run.smithi018.stdout: rgw cache enabled = true 2021-09-17T10:06:31.773 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable ops log = true 2021-09-17T10:06:31.773 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable usage log = true 2021-09-17T10:06:31.774 INFO:teuthology.orchestra.run.smithi018.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-17T10:06:31.774 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-17T10:06:31.774 INFO:teuthology.orchestra.run.smithi018.stdout: debug rgw = 20 2021-09-17T10:06:31.774 INFO:teuthology.orchestra.run.smithi018.stdout: rgw crypt require ssl = False 2021-09-17T10:06:31.775 INFO:teuthology.orchestra.run.smithi018.stdout: rgw crypt s3 kms backend = testing 2021-09-17T10:06:31.775 INFO:teuthology.orchestra.run.smithi018.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-17T10:06:31.775 INFO:teuthology.orchestra.run.smithi018.stdout: rgw curl low speed time = 300 2021-09-17T10:06:31.775 INFO:teuthology.orchestra.run.smithi018.stdout: rgw data log num shards = 4 2021-09-17T10:06:31.775 INFO:teuthology.orchestra.run.smithi018.stdout: rgw md log max shards = 4 2021-09-17T10:06:31.776 INFO:teuthology.orchestra.run.smithi018.stdout: rgw sync log trim interval = 0 2021-09-17T10:06:31.776 INFO:teuthology.orchestra.run.smithi018.stdout: rgw sync meta inject err probability = 0.1 2021-09-17T10:06:31.776 INFO:teuthology.orchestra.run.smithi018.stdout: rgw sync obj etag verify = True 2021-09-17T10:06:31.776 INFO:teuthology.orchestra.run.smithi018.stdout: setgroup = ceph 2021-09-17T10:06:31.776 INFO:teuthology.orchestra.run.smithi018.stdout: setuser = ceph 2021-09-17T10:06:31.777 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.a] 2021-09-17T10:06:31.777 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.0] 2021-09-17T10:06:31.777 INFO:teuthology.orchestra.run.smithi018.stdout: osd_max_omap_entries_per_request = 10 2021-09-17T10:06:31.777 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.1] 2021-09-17T10:06:31.778 INFO:teuthology.orchestra.run.smithi018.stdout: osd_max_omap_entries_per_request = 1000 2021-09-17T10:06:31.778 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.2] 2021-09-17T10:06:31.778 INFO:teuthology.orchestra.run.smithi018.stdout: osd_max_omap_entries_per_request = 10000 2021-09-17T10:06:31.781 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2021-09-17T10:06:31.782 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2021-09-17T10:06:31.782 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-17T10:06:31.782 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2021-09-17T10:06:31.782 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.783 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2021-09-17T10:06:31.783 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.783 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2021-09-17T10:06:31.783 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.784 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2021-09-17T10:06:31.784 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2021-09-17T10:06:31.784 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.784 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2021-09-17T10:06:31.784 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2021-09-17T10:06:31.785 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.785 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-17T10:06:31.785 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2021-09-17T10:06:31.785 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.786 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2021-09-17T10:06:31.786 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-17T10:06:31.786 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.786 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2021-09-17T10:06:31.786 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.787 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2021-09-17T10:06:31.787 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2021-09-17T10:06:31.787 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2021-09-17T10:06:31.787 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2021-09-17T10:06:31.788 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2021-09-17T10:06:31.788 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2021-09-17T10:06:31.788 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2021-09-17T10:06:31.788 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-17T10:06:31.788 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-17T10:06:31.789 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2021-09-17T10:06:31.789 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.789 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-17T10:06:31.789 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.790 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2021-09-17T10:06:31.790 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.790 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2021-09-17T10:06:31.790 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.791 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2021-09-17T10:06:31.791 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2021-09-17T10:06:31.791 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2021-09-17T10:06:31.791 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = 172.21.15.50 2021-09-17T10:06:31.791 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 5e649385-aff9-4de1-8a2e-b2ceb667c16c 2021-09-17T10:06:31.792 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.792 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2021-09-17T10:06:31.792 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2021-09-17T10:06:31.792 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.792 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2021-09-17T10:06:31.793 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2021-09-17T10:06:31.793 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.793 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2021-09-17T10:06:31.793 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2021-09-17T10:06:31.794 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.794 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2021-09-17T10:06:31.794 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2021-09-17T10:06:31.794 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2021-09-17T10:06:31.795 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.795 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2021-09-17T10:06:31.795 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2021-09-17T10:06:31.795 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.795 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2021-09-17T10:06:31.796 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.796 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2021-09-17T10:06:31.796 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.796 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2021-09-17T10:06:31.796 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.797 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2021-09-17T10:06:31.797 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2021-09-17T10:06:31.797 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.797 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2021-09-17T10:06:31.798 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2021-09-17T10:06:31.798 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2021-09-17T10:06:31.798 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2021-09-17T10:06:31.798 INFO:teuthology.orchestra.run.smithi050.stdout: rocksdb delete range threshold = 0 2021-09-17T10:06:31.798 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.799 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2021-09-17T10:06:31.799 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2021-09-17T10:06:31.799 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2021-09-17T10:06:31.799 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2021-09-17T10:06:31.800 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2021-09-17T10:06:31.800 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2021-09-17T10:06:31.800 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2021-09-17T10:06:31.800 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2021-09-17T10:06:31.800 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.801 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2021-09-17T10:06:31.801 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2021-09-17T10:06:31.801 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2021-09-17T10:06:31.801 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2021-09-17T10:06:31.801 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2021-09-17T10:06:31.802 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2021-09-17T10:06:31.802 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2021-09-17T10:06:31.802 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2021-09-17T10:06:31.802 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2021-09-17T10:06:31.803 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2021-09-17T10:06:31.803 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2021-09-17T10:06:31.803 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.803 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-17T10:06:31.803 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2021-09-17T10:06:31.804 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2021-09-17T10:06:31.804 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.804 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2021-09-17T10:06:31.804 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-17T10:06:31.805 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-17T10:06:31.805 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:31.805 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2021-09-17T10:06:31.805 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2021-09-17T10:06:31.805 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2021-09-17T10:06:31.806 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2021-09-17T10:06:31.806 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-17T10:06:31.806 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-17T10:06:31.806 INFO:teuthology.orchestra.run.smithi050.stdout: debug rgw = 20 2021-09-17T10:06:31.807 INFO:teuthology.orchestra.run.smithi050.stdout: rgw crypt require ssl = False 2021-09-17T10:06:31.807 INFO:teuthology.orchestra.run.smithi050.stdout: rgw crypt s3 kms backend = testing 2021-09-17T10:06:31.807 INFO:teuthology.orchestra.run.smithi050.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-17T10:06:31.807 INFO:teuthology.orchestra.run.smithi050.stdout: rgw curl low speed time = 300 2021-09-17T10:06:31.807 INFO:teuthology.orchestra.run.smithi050.stdout: rgw data log num shards = 4 2021-09-17T10:06:31.808 INFO:teuthology.orchestra.run.smithi050.stdout: rgw md log max shards = 4 2021-09-17T10:06:31.808 INFO:teuthology.orchestra.run.smithi050.stdout: rgw sync log trim interval = 0 2021-09-17T10:06:31.808 INFO:teuthology.orchestra.run.smithi050.stdout: rgw sync meta inject err probability = 0.1 2021-09-17T10:06:31.808 INFO:teuthology.orchestra.run.smithi050.stdout: rgw sync obj etag verify = True 2021-09-17T10:06:31.809 INFO:teuthology.orchestra.run.smithi050.stdout: setgroup = ceph 2021-09-17T10:06:31.809 INFO:teuthology.orchestra.run.smithi050.stdout: setuser = ceph 2021-09-17T10:06:31.809 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2021-09-17T10:06:31.809 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2021-09-17T10:06:31.809 INFO:teuthology.orchestra.run.smithi050.stdout: osd_max_omap_entries_per_request = 10 2021-09-17T10:06:31.810 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2021-09-17T10:06:31.810 INFO:teuthology.orchestra.run.smithi050.stdout: osd_max_omap_entries_per_request = 1000 2021-09-17T10:06:31.810 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2021-09-17T10:06:31.810 INFO:teuthology.orchestra.run.smithi050.stdout: osd_max_omap_entries_per_request = 10000 2021-09-17T10:06:31.811 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-09-17T10:06:31.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-09-17T10:06:31.860 INFO:tasks.ceph:Copying monmap to all nodes... 2021-09-17T10:06:31.860 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:31.860 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2021-09-17T10:06:31.868 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:31.868 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2021-09-17T10:06:31.875 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi018.front.sepia.ceph.com 2021-09-17T10:06:31.875 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:31.875 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/c2.keyring 2021-09-17T10:06:31.876 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-17T10:06:31.903 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2021-09-17T10:06:31.904 DEBUG:teuthology.orchestra.run.smithi018:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-09-17T10:06:31.952 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2021-09-17T10:06:31.952 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:31.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/c2.keyring 2021-09-17T10:06:31.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-17T10:06:31.980 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:31.980 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-09-17T10:06:32.028 INFO:tasks.ceph:Setting up mon nodes... 2021-09-17T10:06:32.029 INFO:tasks.ceph:Setting up mgr nodes... 2021-09-17T10:06:32.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-09-17T10:06:32.072 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-09-17T10:06:32.075 INFO:tasks.ceph:Setting up mds nodes... 2021-09-17T10:06:32.075 INFO:tasks.ceph_client:Setting up client nodes... 2021-09-17T10:06:32.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-09-17T10:06:32.109 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/c2.client.0.keyring 2021-09-17T10:06:32.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-09-17T10:06:32.198 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/c2.client.1.keyring 2021-09-17T10:06:32.210 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-09-17T10:06:32.211 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-17T10:06:32.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-09-17T10:06:32.223 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T10:06:32.223 INFO:tasks.ceph:role: c2.osd.0 2021-09-17T10:06:32.224 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2021-09-17T10:06:32.224 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-09-17T10:06:32.309 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-09-17T10:06:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-17T10:06:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-17T10:06:32.309 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2021-09-17T10:06:32.310 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-17T10:06:32.310 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2021-09-17T10:06:32.310 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-17T10:06:32.310 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-17T10:06:32.311 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-17T10:06:32.311 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-17T10:06:32.312 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2021-09-17T10:06:32.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-09-17T10:06:32.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-09-17T10:06:32.386 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: /sbin/restorecon: command not found 2021-09-17T10:06:32.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T10:06:32.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-09-17T10:06:32.399 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T10:06:32.399 INFO:tasks.ceph:role: c2.osd.1 2021-09-17T10:06:32.400 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2021-09-17T10:06:32.400 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-09-17T10:06:32.443 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-09-17T10:06:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-17T10:06:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-17T10:06:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2021-09-17T10:06:32.444 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-17T10:06:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2021-09-17T10:06:32.444 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-17T10:06:32.445 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-17T10:06:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-17T10:06:32.445 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-17T10:06:32.446 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2021-09-17T10:06:32.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-09-17T10:06:32.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-09-17T10:06:32.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T10:06:32.519 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: /sbin/restorecon: command not found 2021-09-17T10:06:32.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-09-17T10:06:32.532 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-17T10:06:32.533 INFO:tasks.ceph:role: c2.osd.2 2021-09-17T10:06:32.533 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2021-09-17T10:06:32.534 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-09-17T10:06:32.618 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-09-17T10:06:32.618 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-17T10:06:32.619 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-17T10:06:32.619 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2021-09-17T10:06:32.619 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-17T10:06:32.619 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2021-09-17T10:06:32.620 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-17T10:06:32.620 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-17T10:06:32.620 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-17T10:06:32.620 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-17T10:06:32.621 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2021-09-17T10:06:32.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-09-17T10:06:32.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-09-17T10:06:32.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T10:06:32.653 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: /sbin/restorecon: command not found 2021-09-17T10:06:32.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-17T10:06:32.778 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:32.796+0000 7f35e75b4f40 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-09-17T10:06:32.779 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:32.796+0000 7f35e75b4f40 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-09-17T10:06:32.780 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:32.796+0000 7f35e75b4f40 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-17T10:06:32.780 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:32.796+0000 7f35e75b4f40 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-17T10:06:32.780 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:32.796+0000 7f35e75b4f40 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-17T10:06:32.781 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:32.796+0000 7f35e75b4f40 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-09-17T10:06:34.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-09-17T10:06:34.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-17T10:06:35.019 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:35.036+0000 7fa5481ecf40 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-09-17T10:06:35.020 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:35.040+0000 7fa5481ecf40 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-09-17T10:06:35.020 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:35.040+0000 7fa5481ecf40 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-17T10:06:35.020 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:35.040+0000 7fa5481ecf40 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-17T10:06:35.021 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:35.040+0000 7fa5481ecf40 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-17T10:06:35.021 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:35.040+0000 7fa5481ecf40 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-09-17T10:06:37.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-09-17T10:06:37.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-17T10:06:37.318 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:37.336+0000 7f875b2b7f40 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-09-17T10:06:37.318 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:37.336+0000 7f875b2b7f40 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-09-17T10:06:37.319 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:37.336+0000 7f875b2b7f40 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-17T10:06:37.319 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:37.336+0000 7f875b2b7f40 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-17T10:06:37.319 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:37.336+0000 7f875b2b7f40 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-17T10:06:37.320 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:06:37.336+0000 7f875b2b7f40 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-09-17T10:06:39.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-09-17T10:06:39.520 INFO:tasks.ceph:Reading keys from all nodes... 2021-09-17T10:06:39.520 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:39.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2021-09-17T10:06:39.532 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:39.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2021-09-17T10:06:39.547 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:39.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2021-09-17T10:06:39.604 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:39.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2021-09-17T10:06:39.620 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:39.621 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2021-09-17T10:06:39.627 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:39.628 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2021-09-17T10:06:39.634 INFO:tasks.ceph:Adding keys to all mons... 2021-09-17T10:06:39.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/c2.keyring 2021-09-17T10:06:39.646 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2021-09-17T10:06:39.647 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCoaERhUB+hBRAASSI5ypaIepIfqwKD3AZnxw== 2021-09-17T10:06:39.647 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2021-09-17T10:06:39.647 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCoaERhjSqlLxAArlbckiVDPUDrLEfQSclyTQ== 2021-09-17T10:06:39.647 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2021-09-17T10:06:39.648 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCraERh5SJ0AhAAYz0CmmAtfctfu4e0tnvS/A== 2021-09-17T10:06:39.648 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2021-09-17T10:06:39.648 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCtaERhj9lBFBAAGjtbb//giYLXJE0VToMgrA== 2021-09-17T10:06:39.648 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2021-09-17T10:06:39.648 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCoaERhGBbXBxAAvp3vjCLCGFJx36Y/RyBIZQ== 2021-09-17T10:06:39.649 INFO:teuthology.orchestra.run.smithi050.stdout:[client.1] 2021-09-17T10:06:39.649 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCoaERhj1UiDRAAmB1pTdiqFJDcsebiqFpqtQ== 2021-09-17T10:06:39.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-09-17T10:06:39.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-17T10:06:39.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-17T10:06:39.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-17T10:06:39.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-17T10:06:39.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-17T10:06:39.947 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-09-17T10:06:39.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-09-17T10:06:40.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-09-17T10:06:40.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-09-17T10:06:40.081 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-09-17T10:06:40.085 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-09-17T10:06:40.086 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-09-17T10:06:40.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-09-17T10:06:40.128 INFO:tasks.ceph.c2.mon.a:Started 2021-09-17T10:06:40.129 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-09-17T10:06:40.129 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-09-17T10:06:40.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-09-17T10:06:40.132 INFO:tasks.ceph.c2.mgr.x:Started 2021-09-17T10:06:40.132 INFO:tasks.ceph:Setting crush tunables to default 2021-09-17T10:06:40.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster c2 osd crush tunables default 2021-09-17T10:06:40.352 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2021-09-17T10:06:40.363 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-09-17T10:06:40.364 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:40.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2021-09-17T10:06:40.380 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:40.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2021-09-17T10:06:40.393 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2021-09-17T10:06:40.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2021-09-17T10:06:40.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster c2 osd new ff03c9da-b048-4c9b-b123-d00df9fb05c0 0 2021-09-17T10:06:40.637 INFO:teuthology.orchestra.run.smithi050.stdout:0 2021-09-17T10:06:40.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster c2 osd new 4ccb2eda-cd7f-43b1-aeb8-df3cc3746035 1 2021-09-17T10:06:40.857 INFO:teuthology.orchestra.run.smithi050.stdout:1 2021-09-17T10:06:40.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster c2 osd new 0f1d3069-deea-41ee-b499-3a6485c91c52 2 2021-09-17T10:06:41.063 INFO:teuthology.orchestra.run.smithi050.stdout:2 2021-09-17T10:06:41.075 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-09-17T10:06:41.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-09-17T10:06:41.078 INFO:tasks.ceph.c2.osd.0:Started 2021-09-17T10:06:41.079 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-09-17T10:06:41.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-09-17T10:06:41.082 INFO:tasks.ceph.c2.osd.1:Started 2021-09-17T10:06:41.082 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-09-17T10:06:41.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-09-17T10:06:41.085 INFO:tasks.ceph.c2.osd.2:Started 2021-09-17T10:06:41.086 INFO:tasks.ceph:Waiting for OSDs to come up 2021-09-17T10:06:41.389 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-17T10:06:41.557 INFO:teuthology.misc.health.smithi050.stdout: 2021-09-17T10:06:41.557 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":5,"fsid":"5e649385-aff9-4de1-8a2e-b2ceb667c16c","created":"2021-09-17T10:06:40.264686+0000","modified":"2021-09-17T10:06:41.082444+0000","last_up_change":"0.000000","last_in_change":"2021-09-17T10:06:41.082444+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ff03c9da-b048-4c9b-b123-d00df9fb05c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ccb2eda-cd7f-43b1-aeb8-df3cc3746035","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0f1d3069-deea-41ee-b499-3a6485c91c52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T10:06:41.567 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-09-17T10:06:41.705 INFO:tasks.ceph.c2.osd.2.smithi050.stderr:2021-09-17T10:06:41.724+0000 7f0f63629f40 -1 Falling back to public interface 2021-09-17T10:06:41.705 INFO:tasks.ceph.c2.osd.0.smithi050.stderr:2021-09-17T10:06:41.724+0000 7fc59e488f40 -1 Falling back to public interface 2021-09-17T10:06:41.706 INFO:tasks.ceph.c2.osd.1.smithi050.stderr:2021-09-17T10:06:41.724+0000 7efd3e683f40 -1 Falling back to public interface 2021-09-17T10:06:41.706 INFO:tasks.ceph.c2.osd.0.smithi050.stderr:2021-09-17T10:06:41.724+0000 7fc59d453700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6803/0: (98) Address already in use 2021-09-17T10:06:44.228 INFO:tasks.ceph.c2.osd.1.smithi050.stderr:2021-09-17T10:06:44.248+0000 7efd3e683f40 -1 osd.1 0 log_to_monitors {default=true} 2021-09-17T10:06:44.431 INFO:tasks.ceph.c2.osd.2.smithi050.stderr:2021-09-17T10:06:44.452+0000 7f0f63629f40 -1 osd.2 0 log_to_monitors {default=true} 2021-09-17T10:06:44.522 INFO:tasks.ceph.c2.osd.0.smithi050.stderr:2021-09-17T10:06:44.540+0000 7fc59e488f40 -1 osd.0 0 log_to_monitors {default=true} 2021-09-17T10:06:46.862 INFO:tasks.ceph.c2.osd.1.smithi050.stderr:2021-09-17T10:06:46.880+0000 7efd3b64a700 -1 osd.1 0 waiting for initial osdmap 2021-09-17T10:06:46.864 INFO:tasks.ceph.c2.osd.1.smithi050.stderr:2021-09-17T10:06:46.884+0000 7efd32bc8700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-17T10:06:46.873 INFO:tasks.ceph.c2.osd.2.smithi050.stderr:2021-09-17T10:06:46.892+0000 7f0f5edcb700 -1 osd.2 0 waiting for initial osdmap 2021-09-17T10:06:46.876 INFO:tasks.ceph.c2.osd.2.smithi050.stderr:2021-09-17T10:06:46.896+0000 7f0f57b6e700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-17T10:06:46.902 INFO:tasks.ceph.c2.osd.0.smithi050.stderr:2021-09-17T10:06:46.920+0000 7fc59b44f700 -1 osd.0 0 waiting for initial osdmap 2021-09-17T10:06:46.905 INFO:tasks.ceph.c2.osd.0.smithi050.stderr:2021-09-17T10:06:46.924+0000 7fc5929cd700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-17T10:06:47.871 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-17T10:06:48.143 INFO:teuthology.misc.health.smithi050.stdout: 2021-09-17T10:06:48.143 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":8,"fsid":"5e649385-aff9-4de1-8a2e-b2ceb667c16c","created":"2021-09-17T10:06:40.264686+0000","modified":"2021-09-17T10:06:47.881357+0000","last_up_change":"2021-09-17T10:06:47.881357+0000","last_in_change":"2021-09-17T10:06:41.082444+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"ff03c9da-b048-4c9b-b123-d00df9fb05c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":14045},{"type":"v1","addr":"172.21.15.50:6808","nonce":14045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":14045},{"type":"v1","addr":"172.21.15.50:6815","nonce":14045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":14045},{"type":"v1","addr":"172.21.15.50:6823","nonce":14045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":14045},{"type":"v1","addr":"172.21.15.50:6820","nonce":14045}]},"public_addr":"172.21.15.50:6808/14045","cluster_addr":"172.21.15.50:6815/14045","heartbeat_back_addr":"172.21.15.50:6823/14045","heartbeat_front_addr":"172.21.15.50:6820/14045","state":["exists","up"]},{"osd":1,"uuid":"4ccb2eda-cd7f-43b1-aeb8-df3cc3746035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":14046},{"type":"v1","addr":"172.21.15.50:6806","nonce":14046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":14046},{"type":"v1","addr":"172.21.15.50:6813","nonce":14046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6819","nonce":14046},{"type":"v1","addr":"172.21.15.50:6821","nonce":14046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":14046},{"type":"v1","addr":"172.21.15.50:6818","nonce":14046}]},"public_addr":"172.21.15.50:6806/14046","cluster_addr":"172.21.15.50:6813/14046","heartbeat_back_addr":"172.21.15.50:6821/14046","heartbeat_front_addr":"172.21.15.50:6818/14046","state":["exists","up"]},{"osd":2,"uuid":"0f1d3069-deea-41ee-b499-3a6485c91c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":14047},{"type":"v1","addr":"172.21.15.50:6801","nonce":14047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":14047},{"type":"v1","addr":"172.21.15.50:6804","nonce":14047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":14047},{"type":"v1","addr":"172.21.15.50:6814","nonce":14047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":14047},{"type":"v1","addr":"172.21.15.50:6809","nonce":14047}]},"public_addr":"172.21.15.50:6801/14047","cluster_addr":"172.21.15.50:6804/14047","heartbeat_back_addr":"172.21.15.50:6814/14047","heartbeat_front_addr":"172.21.15.50:6809/14047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T10:06:48.155 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-09-17T10:06:48.156 INFO:tasks.ceph:Creating RBD pool 2021-09-17T10:06:48.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster c2 osd pool create rbd 8 2021-09-17T10:06:48.876 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'rbd' created 2021-09-17T10:06:48.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-09-17T10:06:49.882 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rbd' on pool 'rbd' 2021-09-17T10:06:49.894 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-09-17T10:06:49.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-17T10:06:49.896 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-09-17T10:06:50.185 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:50.185 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":10,"fsid":"5e649385-aff9-4de1-8a2e-b2ceb667c16c","created":"2021-09-17T10:06:40.264686+0000","modified":"2021-09-17T10:06:49.884748+0000","last_up_change":"2021-09-17T10:06:47.881357+0000","last_in_change":"2021-09-17T10:06:41.082444+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-17T10:06:47.994256+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-09-17T10:06:48.500907+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ff03c9da-b048-4c9b-b123-d00df9fb05c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":14045},{"type":"v1","addr":"172.21.15.50:6808","nonce":14045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":14045},{"type":"v1","addr":"172.21.15.50:6815","nonce":14045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":14045},{"type":"v1","addr":"172.21.15.50:6823","nonce":14045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":14045},{"type":"v1","addr":"172.21.15.50:6820","nonce":14045}]},"public_addr":"172.21.15.50:6808/14045","cluster_addr":"172.21.15.50:6815/14045","heartbeat_back_addr":"172.21.15.50:6823/14045","heartbeat_front_addr":"172.21.15.50:6820/14045","state":["exists","up"]},{"osd":1,"uuid":"4ccb2eda-cd7f-43b1-aeb8-df3cc3746035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":14046},{"type":"v1","addr":"172.21.15.50:6806","nonce":14046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":14046},{"type":"v1","addr":"172.21.15.50:6813","nonce":14046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6819","nonce":14046},{"type":"v1","addr":"172.21.15.50:6821","nonce":14046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":14046},{"type":"v1","addr":"172.21.15.50:6818","nonce":14046}]},"public_addr":"172.21.15.50:6806/14046","cluster_addr":"172.21.15.50:6813/14046","heartbeat_back_addr":"172.21.15.50:6821/14046","heartbeat_front_addr":"172.21.15.50:6818/14046","state":["exists","up"]},{"osd":2,"uuid":"0f1d3069-deea-41ee-b499-3a6485c91c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":14047},{"type":"v1","addr":"172.21.15.50:6801","nonce":14047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":14047},{"type":"v1","addr":"172.21.15.50:6804","nonce":14047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":14047},{"type":"v1","addr":"172.21.15.50:6814","nonce":14047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":14047},{"type":"v1","addr":"172.21.15.50:6809","nonce":14047}]},"public_addr":"172.21.15.50:6801/14047","cluster_addr":"172.21.15.50:6804/14047","heartbeat_back_addr":"172.21.15.50:6814/14047","heartbeat_front_addr":"172.21.15.50:6809/14047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:45.532673+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:45.237132+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:45.464465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T10:06:50.198 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2021-09-17T10:06:47.994256+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-09-17T10:06:48.500907+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-09-17T10:06:50.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get .mgr pg_num 2021-09-17T10:06:50.468 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2021-09-17T10:06:50.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2021-09-17T10:06:50.759 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 8 2021-09-17T10:06:50.773 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-09-17T10:06:50.773 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-09-17T10:06:50.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2021-09-17T10:06:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:51.083 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":13860},{"type":"v1","addr":"172.21.15.50:6825","nonce":13860}]},"active_addr":"172.21.15.50:6825/13860","active_change":"2021-09-17T10:06:43.930484+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":4264289488}]},{"addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":2472522355}]},{"addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":3416421812}]}]}} 2021-09-17T10:06:51.085 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-09-17T10:06:51.086 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-09-17T10:06:51.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-17T10:06:51.349 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:51.350 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":11,"fsid":"5e649385-aff9-4de1-8a2e-b2ceb667c16c","created":"2021-09-17T10:06:40.264686+0000","modified":"2021-09-17T10:06:50.886887+0000","last_up_change":"2021-09-17T10:06:47.881357+0000","last_in_change":"2021-09-17T10:06:41.082444+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-17T10:06:47.994256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-09-17T10:06:48.500907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ff03c9da-b048-4c9b-b123-d00df9fb05c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":14045},{"type":"v1","addr":"172.21.15.50:6808","nonce":14045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":14045},{"type":"v1","addr":"172.21.15.50:6815","nonce":14045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":14045},{"type":"v1","addr":"172.21.15.50:6823","nonce":14045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":14045},{"type":"v1","addr":"172.21.15.50:6820","nonce":14045}]},"public_addr":"172.21.15.50:6808/14045","cluster_addr":"172.21.15.50:6815/14045","heartbeat_back_addr":"172.21.15.50:6823/14045","heartbeat_front_addr":"172.21.15.50:6820/14045","state":["exists","up"]},{"osd":1,"uuid":"4ccb2eda-cd7f-43b1-aeb8-df3cc3746035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":14046},{"type":"v1","addr":"172.21.15.50:6806","nonce":14046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":14046},{"type":"v1","addr":"172.21.15.50:6813","nonce":14046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6819","nonce":14046},{"type":"v1","addr":"172.21.15.50:6821","nonce":14046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":14046},{"type":"v1","addr":"172.21.15.50:6818","nonce":14046}]},"public_addr":"172.21.15.50:6806/14046","cluster_addr":"172.21.15.50:6813/14046","heartbeat_back_addr":"172.21.15.50:6821/14046","heartbeat_front_addr":"172.21.15.50:6818/14046","state":["exists","up"]},{"osd":2,"uuid":"0f1d3069-deea-41ee-b499-3a6485c91c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":14047},{"type":"v1","addr":"172.21.15.50:6801","nonce":14047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":14047},{"type":"v1","addr":"172.21.15.50:6804","nonce":14047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":14047},{"type":"v1","addr":"172.21.15.50:6814","nonce":14047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":14047},{"type":"v1","addr":"172.21.15.50:6809","nonce":14047}]},"public_addr":"172.21.15.50:6801/14047","cluster_addr":"172.21.15.50:6804/14047","heartbeat_back_addr":"172.21.15.50:6814/14047","heartbeat_front_addr":"172.21.15.50:6809/14047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:45.532673+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:45.237132+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:45.464465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T10:06:51.362 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-09-17T10:06:51.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-17T10:06:51.626 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:51.626 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":11,"fsid":"5e649385-aff9-4de1-8a2e-b2ceb667c16c","created":"2021-09-17T10:06:40.264686+0000","modified":"2021-09-17T10:06:50.886887+0000","last_up_change":"2021-09-17T10:06:47.881357+0000","last_in_change":"2021-09-17T10:06:41.082444+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-17T10:06:47.994256+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-09-17T10:06:48.500907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ff03c9da-b048-4c9b-b123-d00df9fb05c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":14045},{"type":"v1","addr":"172.21.15.50:6808","nonce":14045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":14045},{"type":"v1","addr":"172.21.15.50:6815","nonce":14045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":14045},{"type":"v1","addr":"172.21.15.50:6823","nonce":14045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":14045},{"type":"v1","addr":"172.21.15.50:6820","nonce":14045}]},"public_addr":"172.21.15.50:6808/14045","cluster_addr":"172.21.15.50:6815/14045","heartbeat_back_addr":"172.21.15.50:6823/14045","heartbeat_front_addr":"172.21.15.50:6820/14045","state":["exists","up"]},{"osd":1,"uuid":"4ccb2eda-cd7f-43b1-aeb8-df3cc3746035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":14046},{"type":"v1","addr":"172.21.15.50:6806","nonce":14046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":14046},{"type":"v1","addr":"172.21.15.50:6813","nonce":14046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6819","nonce":14046},{"type":"v1","addr":"172.21.15.50:6821","nonce":14046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":14046},{"type":"v1","addr":"172.21.15.50:6818","nonce":14046}]},"public_addr":"172.21.15.50:6806/14046","cluster_addr":"172.21.15.50:6813/14046","heartbeat_back_addr":"172.21.15.50:6821/14046","heartbeat_front_addr":"172.21.15.50:6818/14046","state":["exists","up"]},{"osd":2,"uuid":"0f1d3069-deea-41ee-b499-3a6485c91c52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":14047},{"type":"v1","addr":"172.21.15.50:6801","nonce":14047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":14047},{"type":"v1","addr":"172.21.15.50:6804","nonce":14047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":14047},{"type":"v1","addr":"172.21.15.50:6814","nonce":14047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":14047},{"type":"v1","addr":"172.21.15.50:6809","nonce":14047}]},"public_addr":"172.21.15.50:6801/14047","cluster_addr":"172.21.15.50:6804/14047","heartbeat_back_addr":"172.21.15.50:6814/14047","heartbeat_front_addr":"172.21.15.50:6809/14047","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:45.532673+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:45.237132+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-17T10:06:45.464465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-17T10:06:51.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2021-09-17T10:06:51.785 INFO:teuthology.orchestra.run.smithi050.stdout:34359738370 2021-09-17T10:06:51.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2021-09-17T10:06:51.932 INFO:teuthology.orchestra.run.smithi050.stdout:34359738371 2021-09-17T10:06:51.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2021-09-17T10:06:52.088 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2021-09-17T10:06:52.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-09-17T10:06:52.358 INFO:teuthology.orchestra.run.smithi050.stdout:34359738370 2021-09-17T10:06:52.372 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.0 2021-09-17T10:06:52.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-09-17T10:06:52.647 INFO:teuthology.orchestra.run.smithi050.stdout:34359738370 2021-09-17T10:06:52.660 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.1 2021-09-17T10:06:53.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-09-17T10:06:53.945 INFO:teuthology.orchestra.run.smithi050.stdout:34359738370 2021-09-17T10:06:53.959 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.1 2021-09-17T10:06:54.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-09-17T10:06:55.248 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2021-09-17T10:06:55.262 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738372 for osd.1 2021-09-17T10:06:55.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2021-09-17T10:06:55.530 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2021-09-17T10:06:55.543 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738373 for osd.2 2021-09-17T10:06:55.543 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-09-17T10:06:55.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-09-17T10:06:55.805 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:55.809 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2021-09-17T10:06:55.819 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2021-09-17T10:06:53.938779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":33148,"kb_used_data":1444,"kb_used_omap":0,"kb_used_meta":31424,"kb_avail":314539652,"statfs":{"total":322122547200,"available":322088603648,"internally_reserved":0,"allocated":1478656,"data_stored":1258225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32178176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.904115+0000","last_change":"2021-09-17T10:06:49.902311+0000","last_active":"2021-09-17T10:06:49.904115+0000","last_peered":"2021-09-17T10:06:49.904115+0000","last_clean":"2021-09-17T10:06:49.904115+0000","last_became_active":"2021-09-17T10:06:49.900919+0000","last_became_peered":"2021-09-17T10:06:49.900919+0000","last_unstale":"2021-09-17T10:06:49.904115+0000","last_undegraded":"2021-09-17T10:06:49.904115+0000","last_fullsized":"2021-09-17T10:06:49.904115+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.904067+0000","last_change":"2021-09-17T10:06:49.902004+0000","last_active":"2021-09-17T10:06:49.904067+0000","last_peered":"2021-09-17T10:06:49.904067+0000","last_clean":"2021-09-17T10:06:49.904067+0000","last_became_active":"2021-09-17T10:06:49.901565+0000","last_became_peered":"2021-09-17T10:06:49.901565+0000","last_unstale":"2021-09-17T10:06:49.904067+0000","last_undegraded":"2021-09-17T10:06:49.904067+0000","last_fullsized":"2021-09-17T10:06:49.904067+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.903289+0000","last_change":"2021-09-17T10:06:49.901871+0000","last_active":"2021-09-17T10:06:49.903289+0000","last_peered":"2021-09-17T10:06:49.903289+0000","last_clean":"2021-09-17T10:06:49.903289+0000","last_became_active":"2021-09-17T10:06:49.900537+0000","last_became_peered":"2021-09-17T10:06:49.900537+0000","last_unstale":"2021-09-17T10:06:49.903289+0000","last_undegraded":"2021-09-17T10:06:49.903289+0000","last_fullsized":"2021-09-17T10:06:49.903289+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.904107+0000","last_change":"2021-09-17T10:06:49.902195+0000","last_active":"2021-09-17T10:06:49.904107+0000","last_peered":"2021-09-17T10:06:49.904107+0000","last_clean":"2021-09-17T10:06:49.904107+0000","last_became_active":"2021-09-17T10:06:49.901401+0000","last_became_peered":"2021-09-17T10:06:49.901401+0000","last_unstale":"2021-09-17T10:06:49.904107+0000","last_undegraded":"2021-09-17T10:06:49.904107+0000","last_fullsized":"2021-09-17T10:06:49.904107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.901240+0000","last_change":"2021-09-17T10:06:49.900009+0000","last_active":"2021-09-17T10:06:49.901240+0000","last_peered":"2021-09-17T10:06:49.901240+0000","last_clean":"2021-09-17T10:06:49.901240+0000","last_became_active":"2021-09-17T10:06:49.899668+0000","last_became_peered":"2021-09-17T10:06:49.899668+0000","last_unstale":"2021-09-17T10:06:49.901240+0000","last_undegraded":"2021-09-17T10:06:49.901240+0000","last_fullsized":"2021-09-17T10:06:49.901240+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.902107+0000","last_change":"2021-09-17T10:06:49.900092+0000","last_active":"2021-09-17T10:06:49.902107+0000","last_peered":"2021-09-17T10:06:49.902107+0000","last_clean":"2021-09-17T10:06:49.902107+0000","last_became_active":"2021-09-17T10:06:49.899698+0000","last_became_peered":"2021-09-17T10:06:49.899698+0000","last_unstale":"2021-09-17T10:06:49.902107+0000","last_undegraded":"2021-09-17T10:06:49.902107+0000","last_fullsized":"2021-09-17T10:06:49.902107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.901107+0000","last_change":"2021-09-17T10:06:49.899863+0000","last_active":"2021-09-17T10:06:49.901107+0000","last_peered":"2021-09-17T10:06:49.901107+0000","last_clean":"2021-09-17T10:06:49.901107+0000","last_became_active":"2021-09-17T10:06:49.899509+0000","last_became_peered":"2021-09-17T10:06:49.899509+0000","last_unstale":"2021-09-17T10:06:49.901107+0000","last_undegraded":"2021-09-17T10:06:49.901107+0000","last_fullsized":"2021-09-17T10:06:49.901107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.902227+0000","last_change":"2021-09-17T10:06:49.899861+0000","last_active":"2021-09-17T10:06:49.902227+0000","last_peered":"2021-09-17T10:06:49.902227+0000","last_clean":"2021-09-17T10:06:49.902227+0000","last_became_active":"2021-09-17T10:06:49.899464+0000","last_became_peered":"2021-09-17T10:06:49.899464+0000","last_unstale":"2021-09-17T10:06:49.902227+0000","last_undegraded":"2021-09-17T10:06:49.902227+0000","last_fullsized":"2021-09-17T10:06:49.902227+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:50.508821+0000","last_change":"2021-09-17T10:06:49.901847+0000","last_active":"2021-09-17T10:06:50.508821+0000","last_peered":"2021-09-17T10:06:50.508821+0000","last_clean":"2021-09-17T10:06:50.508821+0000","last_became_active":"2021-09-17T10:06:49.901322+0000","last_became_peered":"2021-09-17T10:06:49.901322+0000","last_unstale":"2021-09-17T10:06:50.508821+0000","last_undegraded":"2021-09-17T10:06:50.508821+0000","last_fullsized":"2021-09-17T10:06:50.508821+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7588,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850012,"statfs":{"total":107374182400,"available":107366412288,"internally_reserved":0,"allocated":94208,"data_stored":25105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":12780,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":11968,"kb_avail":104844820,"statfs":{"total":107374182400,"available":107361095680,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12255232},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":12780,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":11968,"kb_avail":104844820,"statfs":{"total":107374182400,"available":107361095680,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12255232},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-17T10:06:55.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-09-17T10:06:56.101 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:56.103 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2021-09-17T10:06:56.117 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2021-09-17T10:06:55.939272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":33148,"kb_used_data":1444,"kb_used_omap":0,"kb_used_meta":31424,"kb_avail":314539652,"statfs":{"total":322122547200,"available":322088603648,"internally_reserved":0,"allocated":1478656,"data_stored":1258225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32178176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000492"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.904115+0000","last_change":"2021-09-17T10:06:49.902311+0000","last_active":"2021-09-17T10:06:49.904115+0000","last_peered":"2021-09-17T10:06:49.904115+0000","last_clean":"2021-09-17T10:06:49.904115+0000","last_became_active":"2021-09-17T10:06:49.900919+0000","last_became_peered":"2021-09-17T10:06:49.900919+0000","last_unstale":"2021-09-17T10:06:49.904115+0000","last_undegraded":"2021-09-17T10:06:49.904115+0000","last_fullsized":"2021-09-17T10:06:49.904115+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.904067+0000","last_change":"2021-09-17T10:06:49.902004+0000","last_active":"2021-09-17T10:06:49.904067+0000","last_peered":"2021-09-17T10:06:49.904067+0000","last_clean":"2021-09-17T10:06:49.904067+0000","last_became_active":"2021-09-17T10:06:49.901565+0000","last_became_peered":"2021-09-17T10:06:49.901565+0000","last_unstale":"2021-09-17T10:06:49.904067+0000","last_undegraded":"2021-09-17T10:06:49.904067+0000","last_fullsized":"2021-09-17T10:06:49.904067+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.903289+0000","last_change":"2021-09-17T10:06:49.901871+0000","last_active":"2021-09-17T10:06:49.903289+0000","last_peered":"2021-09-17T10:06:49.903289+0000","last_clean":"2021-09-17T10:06:49.903289+0000","last_became_active":"2021-09-17T10:06:49.900537+0000","last_became_peered":"2021-09-17T10:06:49.900537+0000","last_unstale":"2021-09-17T10:06:49.903289+0000","last_undegraded":"2021-09-17T10:06:49.903289+0000","last_fullsized":"2021-09-17T10:06:49.903289+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.904107+0000","last_change":"2021-09-17T10:06:49.902195+0000","last_active":"2021-09-17T10:06:49.904107+0000","last_peered":"2021-09-17T10:06:49.904107+0000","last_clean":"2021-09-17T10:06:49.904107+0000","last_became_active":"2021-09-17T10:06:49.901401+0000","last_became_peered":"2021-09-17T10:06:49.901401+0000","last_unstale":"2021-09-17T10:06:49.904107+0000","last_undegraded":"2021-09-17T10:06:49.904107+0000","last_fullsized":"2021-09-17T10:06:49.904107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.901240+0000","last_change":"2021-09-17T10:06:49.900009+0000","last_active":"2021-09-17T10:06:49.901240+0000","last_peered":"2021-09-17T10:06:49.901240+0000","last_clean":"2021-09-17T10:06:49.901240+0000","last_became_active":"2021-09-17T10:06:49.899668+0000","last_became_peered":"2021-09-17T10:06:49.899668+0000","last_unstale":"2021-09-17T10:06:49.901240+0000","last_undegraded":"2021-09-17T10:06:49.901240+0000","last_fullsized":"2021-09-17T10:06:49.901240+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.902107+0000","last_change":"2021-09-17T10:06:49.900092+0000","last_active":"2021-09-17T10:06:49.902107+0000","last_peered":"2021-09-17T10:06:49.902107+0000","last_clean":"2021-09-17T10:06:49.902107+0000","last_became_active":"2021-09-17T10:06:49.899698+0000","last_became_peered":"2021-09-17T10:06:49.899698+0000","last_unstale":"2021-09-17T10:06:49.902107+0000","last_undegraded":"2021-09-17T10:06:49.902107+0000","last_fullsized":"2021-09-17T10:06:49.902107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.901107+0000","last_change":"2021-09-17T10:06:49.899863+0000","last_active":"2021-09-17T10:06:49.901107+0000","last_peered":"2021-09-17T10:06:49.901107+0000","last_clean":"2021-09-17T10:06:49.901107+0000","last_became_active":"2021-09-17T10:06:49.899509+0000","last_became_peered":"2021-09-17T10:06:49.899509+0000","last_unstale":"2021-09-17T10:06:49.901107+0000","last_undegraded":"2021-09-17T10:06:49.901107+0000","last_fullsized":"2021-09-17T10:06:49.901107+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:49.902227+0000","last_change":"2021-09-17T10:06:49.899861+0000","last_active":"2021-09-17T10:06:49.902227+0000","last_peered":"2021-09-17T10:06:49.902227+0000","last_clean":"2021-09-17T10:06:49.902227+0000","last_became_active":"2021-09-17T10:06:49.899464+0000","last_became_peered":"2021-09-17T10:06:49.899464+0000","last_unstale":"2021-09-17T10:06:49.902227+0000","last_undegraded":"2021-09-17T10:06:49.902227+0000","last_fullsized":"2021-09-17T10:06:49.902227+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-17T10:06:50.508821+0000","last_change":"2021-09-17T10:06:49.901847+0000","last_active":"2021-09-17T10:06:50.508821+0000","last_peered":"2021-09-17T10:06:50.508821+0000","last_clean":"2021-09-17T10:06:50.508821+0000","last_became_active":"2021-09-17T10:06:49.901322+0000","last_became_peered":"2021-09-17T10:06:49.901322+0000","last_unstale":"2021-09-17T10:06:50.508821+0000","last_undegraded":"2021-09-17T10:06:50.508821+0000","last_fullsized":"2021-09-17T10:06:50.508821+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-17T10:06:48.882945+0000","last_clean_scrub_stamp":"2021-09-17T10:06:48.882945+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7588,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850012,"statfs":{"total":107374182400,"available":107366412288,"internally_reserved":0,"allocated":94208,"data_stored":25105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":12780,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":11968,"kb_avail":104844820,"statfs":{"total":107374182400,"available":107361095680,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12255232},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":12780,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":11968,"kb_avail":104844820,"statfs":{"total":107374182400,"available":107361095680,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12255232},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-17T10:06:56.118 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-09-17T10:06:56.118 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-09-17T10:06:56.118 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-09-17T10:06:56.119 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2021-09-17T10:06:56.439 INFO:teuthology.orchestra.run.smithi050.stdout: 2021-09-17T10:06:56.439 INFO:teuthology.orchestra.run.smithi050.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-09-17T10:06:56.452 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-09-17T10:06:56.453 INFO:teuthology.run_tasks:Running task rgw... 2021-09-17T10:06:56.467 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-09-17T10:06:56.467 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-09-17T10:06:56.468 INFO:tasks.rgw:Creating data pools 2021-09-17T10:06:56.468 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-09-17T10:06:56.468 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-09-17T10:06:57.039 INFO:teuthology.orchestra.run.smithi018.stderr:pool 'default.rgw.buckets.data' created 2021-09-17T10:06:57.054 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-09-17T10:06:58.072 INFO:teuthology.orchestra.run.smithi018.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-17T10:06:58.082 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-09-17T10:06:59.063 INFO:teuthology.orchestra.run.smithi018.stderr:pool 'default.rgw.buckets.index' created 2021-09-17T10:06:59.078 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-09-17T10:07:00.085 INFO:teuthology.orchestra.run.smithi018.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-17T10:07:00.098 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-09-17T10:07:00.098 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-09-17T10:07:00.372 INFO:teuthology.orchestra.run.smithi018.stderr:pool 'default.rgw.buckets.data' already exists 2021-09-17T10:07:00.381 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-09-17T10:07:01.059 INFO:teuthology.orchestra.run.smithi018.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-17T10:07:01.071 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-09-17T10:07:01.346 INFO:teuthology.orchestra.run.smithi018.stderr:pool 'default.rgw.buckets.index' already exists 2021-09-17T10:07:01.356 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-09-17T10:07:02.086 INFO:teuthology.orchestra.run.smithi018.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-17T10:07:02.097 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-09-17T10:07:02.098 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-09-17T10:07:03.254 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'default.rgw.buckets.data' created 2021-09-17T10:07:03.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-09-17T10:07:04.276 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-17T10:07:04.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-09-17T10:07:05.269 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'default.rgw.buckets.index' created 2021-09-17T10:07:05.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-09-17T10:07:06.295 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-17T10:07:06.307 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-09-17T10:07:06.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-09-17T10:07:06.575 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'default.rgw.buckets.data' already exists 2021-09-17T10:07:06.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-09-17T10:07:07.273 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-17T10:07:07.285 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-09-17T10:07:07.566 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'default.rgw.buckets.index' already exists 2021-09-17T10:07:07.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-09-17T10:07:08.282 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-17T10:07:08.294 DEBUG:tasks.rgw:Pools created 2021-09-17T10:07:08.294 INFO:tasks.rgw:Configuring compression type = random 2021-09-17T10:07:08.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-09-17T10:07:08.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-09-17T10:07:08.296 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-09-17T10:07:08.351 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:08.351 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:08.366 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:08.382+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.122 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.138+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.122 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.138+0000 7f68a63d0b00 20 realm 2021-09-17T10:07:10.123 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.138+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.123 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.138+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.124 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.138+0000 7f68a63d0b00 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-17T10:07:10.124 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.138+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.128 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.142+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.128 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.142+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.129 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.142+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.129 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.142+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.129 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.146+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.129 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.146+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.130 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.146+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.130 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.146+0000 7f68a63d0b00 10 Creating default zonegroup 2021-09-17T10:07:10.130 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.146+0000 7f68a63d0b00 20 get_system_obj_state: rctx=0x7ffd4fedc4d0 obj=:.pools.avail state=0x5569b7f2ed70 s->prefetch_data=0 2021-09-17T10:07:10.131 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.146+0000 7f68a63d0b00 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-09-17T10:07:10.148 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.162+0000 7f68a63d0b00 10 WARNING: store->list_zones() returned r=-2 2021-09-17T10:07:10.148 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.162+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.148 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.162+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.151 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.166+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.152 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.166+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.152 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.166+0000 7f68a63d0b00 10 could not read realm id: (2) No such file or directory 2021-09-17T10:07:10.153 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.166+0000 7f68a63d0b00 10 WARNING: failed to set zone as default, r=-22 2021-09-17T10:07:10.153 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.166+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.153 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.166+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.156 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.170+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.157 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T10:07:10.157 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.158 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T10:07:10.158 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 10 Using default name default 2021-09-17T10:07:10.158 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.159 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:10.159 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.160 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T10:07:10.160 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.160 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.161 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.161 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.161 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.161 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.174+0000 7f68a63d0b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:10.183 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.198+0000 7f68a63d0b00 20 RGWRados::pool_iterate: got zonegroup_info.d857b486-2751-47e9-abed-36a11d7dbc8b 2021-09-17T10:07:10.183 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.198+0000 7f68a63d0b00 20 RGWRados::pool_iterate: got zone_info.9e6944f0-d8f6-436e-a93d-f675100b411f 2021-09-17T10:07:10.183 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.198+0000 7f68a63d0b00 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T10:07:10.183 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.198+0000 7f68a63d0b00 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T10:07:10.183 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.198+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.184 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.198+0000 7f68a63d0b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:10.185 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.198+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.185 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.198+0000 7f68a63d0b00 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T10:07:10.185 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.198+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.185 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.198+0000 7f68a63d0b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:10.185 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.198+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.187 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.202+0000 7f68a63d0b00 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T10:07:10.187 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.202+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.187 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.202+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.187 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.202+0000 7f68a63d0b00 20 zone default found 2021-09-17T10:07:10.187 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.202+0000 7f68a63d0b00 1 Realm: () 2021-09-17T10:07:10.188 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.202+0000 7f68a63d0b00 1 ZoneGroup: default (d857b486-2751-47e9-abed-36a11d7dbc8b) 2021-09-17T10:07:10.188 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.202+0000 7f68a63d0b00 1 Zone: default (9e6944f0-d8f6-436e-a93d-f675100b411f) 2021-09-17T10:07:10.188 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.202+0000 7f68a63d0b00 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-17T10:07:10.188 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.202+0000 7f68a63d0b00 20 zonegroup default 2021-09-17T10:07:10.189 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.202+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:10.189 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.202+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:10.189 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:10.202+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:12.157 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:12.174+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:12.157 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:12.174+0000 7f68a63d0b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:12.158 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:12.174+0000 7f68a63d0b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:12.158 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:12.174+0000 7f68a63d0b00 20 started sync module instance, tier type = 2021-09-17T10:07:12.158 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:12.174+0000 7f68a63d0b00 20 started zone id=9e6944f0-d8f6-436e-a93d-f675100b411f (name=default) with tier type = 2021-09-17T10:07:14.194 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 add_watcher() i=0 2021-09-17T10:07:14.194 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 add_watcher() i=1 2021-09-17T10:07:14.195 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 add_watcher() i=2 2021-09-17T10:07:14.195 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 add_watcher() i=3 2021-09-17T10:07:14.195 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 add_watcher() i=4 2021-09-17T10:07:14.195 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 add_watcher() i=5 2021-09-17T10:07:14.195 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 add_watcher() i=6 2021-09-17T10:07:14.197 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 add_watcher() i=7 2021-09-17T10:07:14.197 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 2 all 8 watchers are set, enabling cache 2021-09-17T10:07:14.198 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2021-09-17T10:07:14.198 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2021-09-17T10:07:14.198 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T10:07:14.198 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2021-09-17T10:07:14.199 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2021-09-17T10:07:14.199 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T10:07:14.199 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2021-09-17T10:07:14.199 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2021-09-17T10:07:14.200 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T10:07:14.200 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2021-09-17T10:07:14.200 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2021-09-17T10:07:14.200 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T10:07:14.200 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2021-09-17T10:07:14.201 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.210+0000 7f68a63d0b00 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2021-09-17T10:07:14.201 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.214+0000 7f68a63d0b00 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T10:07:14.202 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.218+0000 7f68757e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T10:07:14.202 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.218+0000 7f68a63d0b00 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T10:07:14.202 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:14.218+0000 7f68a63d0b00 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T10:07:18.174 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.188+0000 7f68a63d0b00 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-09-17T10:07:18.174 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.188+0000 7f68a63d0b00 5 note: GC not initialized 2021-09-17T10:07:18.177 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.192+0000 7f67d7fff700 20 reqs_thread_entry: start 2021-09-17T10:07:18.178 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.192+0000 7f67d5ffb700 20 reqs_thread_entry: start 2021-09-17T10:07:18.178 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.192+0000 7f68a63d0b00 20 init_complete bucket index max shards: 11 2021-09-17T10:07:18.179 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.192+0000 7f68a63d0b00 10 rgw notify: Started notification manager with: 1 workers 2021-09-17T10:07:18.179 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.192+0000 7f67c3fff700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 17 10:07:48 2021 2021-09-17T10:07:18.179 INFO:teuthology.orchestra.run.smithi018.stderr: 2021-09-17T10:07:18.182 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.196+0000 7f67c37fe700 20 reqs_thread_entry: start 2021-09-17T10:07:18.206 INFO:teuthology.orchestra.run.smithi018.stdout:[] 2021-09-17T10:07:18.206 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.220+0000 7f68a63d0b00 20 remove_watcher() i=0 2021-09-17T10:07:18.206 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.220+0000 7f68a63d0b00 2 removed watcher, disabling cache 2021-09-17T10:07:18.207 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.224+0000 7f68a63d0b00 20 remove_watcher() i=1 2021-09-17T10:07:18.209 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.224+0000 7f68a63d0b00 20 remove_watcher() i=2 2021-09-17T10:07:18.210 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.224+0000 7f68a63d0b00 20 remove_watcher() i=3 2021-09-17T10:07:18.212 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.228+0000 7f68a63d0b00 20 remove_watcher() i=4 2021-09-17T10:07:18.214 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.228+0000 7f68a63d0b00 20 remove_watcher() i=5 2021-09-17T10:07:18.216 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.232+0000 7f68a63d0b00 20 remove_watcher() i=6 2021-09-17T10:07:18.217 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.232+0000 7f68a63d0b00 20 remove_watcher() i=7 2021-09-17T10:07:18.219 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.236+0000 7f68757e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T10:07:18.228 DEBUG:tasks.util.rgw: json result: [] 2021-09-17T10:07:18.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T10:07:18.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T10:07:18.229 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-17T10:07:18.270 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:18.271 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:18.285 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.300+0000 7f47283acb00 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T10:07:18.285 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.300+0000 7f47283acb00 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T10:07:18.291 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.304+0000 7f47283acb00 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-17T10:07:18.291 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.304+0000 7f466f7fe700 20 reqs_thread_entry: start 2021-09-17T10:07:18.291 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.308+0000 7f47283acb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.293 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.308+0000 7f47283acb00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:18.293 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.308+0000 7f47283acb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.293 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.308+0000 7f47283acb00 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T10:07:18.317 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.332+0000 7f47283acb00 20 RGWRados::pool_iterate: got zonegroup_info.d857b486-2751-47e9-abed-36a11d7dbc8b 2021-09-17T10:07:18.318 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.332+0000 7f47283acb00 20 RGWRados::pool_iterate: got zone_info.9e6944f0-d8f6-436e-a93d-f675100b411f 2021-09-17T10:07:18.318 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.332+0000 7f47283acb00 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T10:07:18.318 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.332+0000 7f47283acb00 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T10:07:18.318 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.332+0000 7f47283acb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.319 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.332+0000 7f47283acb00 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T10:07:18.320 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.332+0000 7f47283acb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.320 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.336+0000 7f47283acb00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:18.320 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.336+0000 7f47283acb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.321 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.336+0000 7f47283acb00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:18.321 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.336+0000 7f47283acb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.321 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.336+0000 7f47283acb00 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T10:07:18.330 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:07:18.330 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "9e6944f0-d8f6-436e-a93d-f675100b411f", 2021-09-17T10:07:18.331 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "default", 2021-09-17T10:07:18.331 INFO:teuthology.orchestra.run.smithi018.stdout: "domain_root": "default.rgw.meta:root", 2021-09-17T10:07:18.331 INFO:teuthology.orchestra.run.smithi018.stdout: "control_pool": "default.rgw.control", 2021-09-17T10:07:18.331 INFO:teuthology.orchestra.run.smithi018.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-17T10:07:18.331 INFO:teuthology.orchestra.run.smithi018.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-17T10:07:18.332 INFO:teuthology.orchestra.run.smithi018.stdout: "log_pool": "default.rgw.log", 2021-09-17T10:07:18.332 INFO:teuthology.orchestra.run.smithi018.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-17T10:07:18.332 INFO:teuthology.orchestra.run.smithi018.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-17T10:07:18.332 INFO:teuthology.orchestra.run.smithi018.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-17T10:07:18.333 INFO:teuthology.orchestra.run.smithi018.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-17T10:07:18.333 INFO:teuthology.orchestra.run.smithi018.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-17T10:07:18.333 INFO:teuthology.orchestra.run.smithi018.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-17T10:07:18.333 INFO:teuthology.orchestra.run.smithi018.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-17T10:07:18.333 INFO:teuthology.orchestra.run.smithi018.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-17T10:07:18.334 INFO:teuthology.orchestra.run.smithi018.stdout: "otp_pool": "default.rgw.otp", 2021-09-17T10:07:18.334 INFO:teuthology.orchestra.run.smithi018.stdout: "system_key": { 2021-09-17T10:07:18.334 INFO:teuthology.orchestra.run.smithi018.stdout: "access_key": "", 2021-09-17T10:07:18.334 INFO:teuthology.orchestra.run.smithi018.stdout: "secret_key": "" 2021-09-17T10:07:18.334 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:18.335 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_pools": [ 2021-09-17T10:07:18.335 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:07:18.335 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "default-placement", 2021-09-17T10:07:18.335 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:07:18.336 INFO:teuthology.orchestra.run.smithi018.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-17T10:07:18.336 INFO:teuthology.orchestra.run.smithi018.stdout: "storage_classes": { 2021-09-17T10:07:18.336 INFO:teuthology.orchestra.run.smithi018.stdout: "STANDARD": { 2021-09-17T10:07:18.336 INFO:teuthology.orchestra.run.smithi018.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-17T10:07:18.336 INFO:teuthology.orchestra.run.smithi018.stdout: "compression_type": "random" 2021-09-17T10:07:18.337 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:18.337 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:18.337 INFO:teuthology.orchestra.run.smithi018.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-17T10:07:18.337 INFO:teuthology.orchestra.run.smithi018.stdout: "index_type": 0 2021-09-17T10:07:18.338 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:18.338 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:18.338 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:07:18.338 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "", 2021-09-17T10:07:18.339 INFO:teuthology.orchestra.run.smithi018.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-17T10:07:18.339 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:07:18.340 DEBUG:tasks.util.rgw: json result: {'id': '9e6944f0-d8f6-436e-a93d-f675100b411f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-17T10:07:18.340 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-09-17T10:07:18.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-09-17T10:07:18.341 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-09-17T10:07:18.378 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:18.379 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:18.393 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.408+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.395 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.412+0000 7f5958cd4b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:18.395 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.412+0000 7f5958cd4b00 20 realm 2021-09-17T10:07:18.396 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.412+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.397 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.412+0000 7f5958cd4b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:18.397 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.412+0000 7f5958cd4b00 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-17T10:07:18.397 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.412+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.397 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.412+0000 7f5958cd4b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:18.398 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.412+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.398 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.412+0000 7f5958cd4b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:18.398 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.412+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.399 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.412+0000 7f5958cd4b00 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T10:07:18.399 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.412+0000 7f5958cd4b00 20 searching for the correct realm 2021-09-17T10:07:18.422 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.436+0000 7f5958cd4b00 20 RGWRados::pool_iterate: got zonegroup_info.d857b486-2751-47e9-abed-36a11d7dbc8b 2021-09-17T10:07:18.422 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.436+0000 7f5958cd4b00 20 RGWRados::pool_iterate: got zone_info.9e6944f0-d8f6-436e-a93d-f675100b411f 2021-09-17T10:07:18.423 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.436+0000 7f5958cd4b00 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T10:07:18.423 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.436+0000 7f5958cd4b00 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T10:07:18.423 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.436+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.424 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.436+0000 7f5958cd4b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:18.424 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.436+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.425 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.440+0000 7f5958cd4b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:18.425 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.440+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.425 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.440+0000 7f5958cd4b00 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T10:07:18.425 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.440+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.426 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.440+0000 7f5958cd4b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:18.426 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.440+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.427 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.440+0000 7f5958cd4b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:18.427 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.440+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.427 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.440+0000 7f5958cd4b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:18.451 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.464+0000 7f5958cd4b00 20 RGWRados::pool_iterate: got zonegroup_info.d857b486-2751-47e9-abed-36a11d7dbc8b 2021-09-17T10:07:18.451 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.464+0000 7f5958cd4b00 20 RGWRados::pool_iterate: got zone_info.9e6944f0-d8f6-436e-a93d-f675100b411f 2021-09-17T10:07:18.451 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.464+0000 7f5958cd4b00 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T10:07:18.452 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.464+0000 7f5958cd4b00 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T10:07:18.452 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.464+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.453 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:18.453 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.453 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T10:07:18.454 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.454 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:18.454 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.455 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T10:07:18.455 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.456 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:18.456 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 20 zone default found 2021-09-17T10:07:18.456 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 1 Realm: () 2021-09-17T10:07:18.456 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 1 ZoneGroup: default (d857b486-2751-47e9-abed-36a11d7dbc8b) 2021-09-17T10:07:18.457 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 1 Zone: default (9e6944f0-d8f6-436e-a93d-f675100b411f) 2021-09-17T10:07:18.457 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-17T10:07:18.457 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 20 zonegroup default 2021-09-17T10:07:18.457 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.468+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.458 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.472+0000 7f5958cd4b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:18.458 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.472+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.459 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.472+0000 7f5958cd4b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:18.459 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.472+0000 7f5958cd4b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.459 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.472+0000 7f5958cd4b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:18.459 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.472+0000 7f5958cd4b00 20 started sync module instance, tier type = 2021-09-17T10:07:18.460 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.472+0000 7f5958cd4b00 20 started zone id=9e6944f0-d8f6-436e-a93d-f675100b411f (name=default) with tier type = 2021-09-17T10:07:18.475 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.488+0000 7f5958cd4b00 20 add_watcher() i=0 2021-09-17T10:07:18.531 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.488+0000 7f5958cd4b00 20 add_watcher() i=1 2021-09-17T10:07:18.532 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.488+0000 7f5958cd4b00 20 add_watcher() i=2 2021-09-17T10:07:18.532 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.488+0000 7f5958cd4b00 20 add_watcher() i=3 2021-09-17T10:07:18.532 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.488+0000 7f5958cd4b00 20 add_watcher() i=4 2021-09-17T10:07:18.532 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.488+0000 7f5958cd4b00 20 add_watcher() i=5 2021-09-17T10:07:18.533 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.492+0000 7f5958cd4b00 20 add_watcher() i=6 2021-09-17T10:07:18.534 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.492+0000 7f5958cd4b00 20 add_watcher() i=7 2021-09-17T10:07:18.534 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.492+0000 7f5958cd4b00 2 all 8 watchers are set, enabling cache 2021-09-17T10:07:18.535 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.492+0000 7f5958cd4b00 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T10:07:18.535 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.492+0000 7f5958cd4b00 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T10:07:18.535 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.496+0000 7f58a3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T10:07:18.535 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.500+0000 7f5958cd4b00 5 note: GC not initialized 2021-09-17T10:07:18.536 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.504+0000 7f5892ffd700 20 reqs_thread_entry: start 2021-09-17T10:07:18.536 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.504+0000 7f5890ff9700 20 reqs_thread_entry: start 2021-09-17T10:07:18.536 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.504+0000 7f5958cd4b00 20 init_complete bucket index max shards: 11 2021-09-17T10:07:18.536 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.504+0000 7f5958cd4b00 10 rgw notify: Started notification manager with: 1 workers 2021-09-17T10:07:18.537 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.504+0000 7f588a7fc700 20 reqs_thread_entry: start 2021-09-17T10:07:18.537 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.504+0000 7f588affd700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 17 10:07:48 2021 2021-09-17T10:07:18.537 INFO:teuthology.orchestra.run.smithi018.stderr: 2021-09-17T10:07:18.537 INFO:teuthology.orchestra.run.smithi018.stdout:[] 2021-09-17T10:07:18.539 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.528+0000 7f5958cd4b00 20 remove_watcher() i=0 2021-09-17T10:07:18.539 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.528+0000 7f5958cd4b00 2 removed watcher, disabling cache 2021-09-17T10:07:18.539 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.528+0000 7f5958cd4b00 20 remove_watcher() i=1 2021-09-17T10:07:18.540 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.532+0000 7f5958cd4b00 20 remove_watcher() i=2 2021-09-17T10:07:18.540 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.536+0000 7f5958cd4b00 20 remove_watcher() i=3 2021-09-17T10:07:18.540 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.536+0000 7f5958cd4b00 20 remove_watcher() i=4 2021-09-17T10:07:18.540 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.540+0000 7f5958cd4b00 20 remove_watcher() i=5 2021-09-17T10:07:18.540 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.540+0000 7f5958cd4b00 20 remove_watcher() i=6 2021-09-17T10:07:18.541 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.544+0000 7f5958cd4b00 20 remove_watcher() i=7 2021-09-17T10:07:18.541 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.544+0000 7f58a3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T10:07:18.542 DEBUG:tasks.util.rgw: json result: [] 2021-09-17T10:07:18.542 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T10:07:18.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T10:07:18.543 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-17T10:07:18.586 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:18.587 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:18.601 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.616+0000 7f728e421b00 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T10:07:18.601 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.616+0000 7f728e421b00 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T10:07:18.607 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.624+0000 7f720d7fa700 20 reqs_thread_entry: start 2021-09-17T10:07:18.608 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.624+0000 7f728e421b00 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-17T10:07:18.608 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.624+0000 7f728e421b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.609 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.624+0000 7f728e421b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:18.609 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.624+0000 7f728e421b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.610 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.624+0000 7f728e421b00 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T10:07:18.633 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.648+0000 7f728e421b00 20 RGWRados::pool_iterate: got zonegroup_info.d857b486-2751-47e9-abed-36a11d7dbc8b 2021-09-17T10:07:18.633 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.648+0000 7f728e421b00 20 RGWRados::pool_iterate: got zone_info.9e6944f0-d8f6-436e-a93d-f675100b411f 2021-09-17T10:07:18.633 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.648+0000 7f728e421b00 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T10:07:18.633 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.648+0000 7f728e421b00 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T10:07:18.634 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.648+0000 7f728e421b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.635 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.648+0000 7f728e421b00 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T10:07:18.635 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.648+0000 7f728e421b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.635 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.648+0000 7f728e421b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:18.636 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.648+0000 7f728e421b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.636 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.648+0000 7f728e421b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:18.636 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.648+0000 7f728e421b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:18.636 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:18.652+0000 7f728e421b00 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T10:07:18.645 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:07:18.646 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "9e6944f0-d8f6-436e-a93d-f675100b411f", 2021-09-17T10:07:18.646 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "default", 2021-09-17T10:07:18.646 INFO:teuthology.orchestra.run.smithi018.stdout: "domain_root": "default.rgw.meta:root", 2021-09-17T10:07:18.647 INFO:teuthology.orchestra.run.smithi018.stdout: "control_pool": "default.rgw.control", 2021-09-17T10:07:18.647 INFO:teuthology.orchestra.run.smithi018.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-17T10:07:18.647 INFO:teuthology.orchestra.run.smithi018.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-17T10:07:18.647 INFO:teuthology.orchestra.run.smithi018.stdout: "log_pool": "default.rgw.log", 2021-09-17T10:07:18.648 INFO:teuthology.orchestra.run.smithi018.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-17T10:07:18.648 INFO:teuthology.orchestra.run.smithi018.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-17T10:07:18.648 INFO:teuthology.orchestra.run.smithi018.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-17T10:07:18.648 INFO:teuthology.orchestra.run.smithi018.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-17T10:07:18.649 INFO:teuthology.orchestra.run.smithi018.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-17T10:07:18.649 INFO:teuthology.orchestra.run.smithi018.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-17T10:07:18.649 INFO:teuthology.orchestra.run.smithi018.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-17T10:07:18.649 INFO:teuthology.orchestra.run.smithi018.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-17T10:07:18.650 INFO:teuthology.orchestra.run.smithi018.stdout: "otp_pool": "default.rgw.otp", 2021-09-17T10:07:18.650 INFO:teuthology.orchestra.run.smithi018.stdout: "system_key": { 2021-09-17T10:07:18.650 INFO:teuthology.orchestra.run.smithi018.stdout: "access_key": "", 2021-09-17T10:07:18.650 INFO:teuthology.orchestra.run.smithi018.stdout: "secret_key": "" 2021-09-17T10:07:18.651 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:18.651 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_pools": [ 2021-09-17T10:07:18.651 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:07:18.651 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "default-placement", 2021-09-17T10:07:18.651 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:07:18.652 INFO:teuthology.orchestra.run.smithi018.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-17T10:07:18.652 INFO:teuthology.orchestra.run.smithi018.stdout: "storage_classes": { 2021-09-17T10:07:18.652 INFO:teuthology.orchestra.run.smithi018.stdout: "STANDARD": { 2021-09-17T10:07:18.652 INFO:teuthology.orchestra.run.smithi018.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-17T10:07:18.653 INFO:teuthology.orchestra.run.smithi018.stdout: "compression_type": "random" 2021-09-17T10:07:18.653 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:18.653 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:18.653 INFO:teuthology.orchestra.run.smithi018.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-17T10:07:18.653 INFO:teuthology.orchestra.run.smithi018.stdout: "index_type": 0 2021-09-17T10:07:18.654 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:18.654 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:18.654 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:07:18.654 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "", 2021-09-17T10:07:18.655 INFO:teuthology.orchestra.run.smithi018.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-17T10:07:18.655 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:07:18.656 DEBUG:tasks.util.rgw: json result: {'id': '9e6944f0-d8f6-436e-a93d-f675100b411f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-17T10:07:18.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-09-17T10:07:18.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-09-17T10:07:18.657 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-09-17T10:07:18.715 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:18.716 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:18.732 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:18.752+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.310 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.332+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.311 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.332+0000 7fcefd695b00 20 realm 2021-09-17T10:07:20.311 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.332+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.311 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.332+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.312 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.332+0000 7fcefd695b00 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-17T10:07:20.312 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.332+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.312 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.332+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.313 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.332+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.318 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.340+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.318 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.340+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.318 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.340+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.319 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.340+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.319 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.340+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.320 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.340+0000 7fcefd695b00 10 Creating default zonegroup 2021-09-17T10:07:20.320 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.340+0000 7fcefd695b00 20 get_system_obj_state: rctx=0x7fffb01f5270 obj=:.pools.avail state=0x55a17082e670 s->prefetch_data=0 2021-09-17T10:07:20.320 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.340+0000 7fcefd695b00 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-09-17T10:07:20.337 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.356+0000 7fcefd695b00 10 WARNING: store->list_zones() returned r=-2 2021-09-17T10:07:20.337 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.356+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.337 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.356+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.340 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.360+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.341 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.360+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.342 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.360+0000 7fcefd695b00 10 could not read realm id: (2) No such file or directory 2021-09-17T10:07:20.342 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.360+0000 7fcefd695b00 10 WARNING: failed to set zone as default, r=-22 2021-09-17T10:07:20.342 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.360+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.342 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.360+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.345 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.364+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.346 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T10:07:20.346 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.347 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T10:07:20.347 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 10 Using default name default 2021-09-17T10:07:20.347 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.348 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:20.348 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.349 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T10:07:20.349 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.350 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.350 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.350 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.350 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.351 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.368+0000 7fcefd695b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:20.370 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 RGWRados::pool_iterate: got zone_info.94df82a5-cc4b-4217-94b9-aec93a1b010e 2021-09-17T10:07:20.371 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 RGWRados::pool_iterate: got zonegroup_info.c34e6d9b-21b3-4d8d-bda2-2d08ebd7d572 2021-09-17T10:07:20.371 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T10:07:20.371 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T10:07:20.371 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.372 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:20.372 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.373 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T10:07:20.373 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.374 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:20.374 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.374 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T10:07:20.375 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.392+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.375 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.396+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.375 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.396+0000 7fcefd695b00 20 zone default found 2021-09-17T10:07:20.375 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.396+0000 7fcefd695b00 1 Realm: () 2021-09-17T10:07:20.376 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.396+0000 7fcefd695b00 1 ZoneGroup: default (c34e6d9b-21b3-4d8d-bda2-2d08ebd7d572) 2021-09-17T10:07:20.376 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.396+0000 7fcefd695b00 1 Zone: default (94df82a5-cc4b-4217-94b9-aec93a1b010e) 2021-09-17T10:07:20.376 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.396+0000 7fcefd695b00 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-17T10:07:20.376 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.396+0000 7fcefd695b00 20 zonegroup default 2021-09-17T10:07:20.376 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.396+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:20.377 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.396+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:20.377 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:20.396+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:22.333 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:22.351+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:22.334 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:22.351+0000 7fcefd695b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:22.334 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:22.351+0000 7fcefd695b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:22.335 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:22.351+0000 7fcefd695b00 20 started sync module instance, tier type = 2021-09-17T10:07:22.335 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:22.351+0000 7fcefd695b00 20 started zone id=94df82a5-cc4b-4217-94b9-aec93a1b010e (name=default) with tier type = 2021-09-17T10:07:24.364 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 add_watcher() i=0 2021-09-17T10:07:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 add_watcher() i=1 2021-09-17T10:07:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 add_watcher() i=2 2021-09-17T10:07:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 add_watcher() i=3 2021-09-17T10:07:24.365 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 add_watcher() i=4 2021-09-17T10:07:24.366 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 add_watcher() i=5 2021-09-17T10:07:24.366 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 add_watcher() i=6 2021-09-17T10:07:24.367 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 add_watcher() i=7 2021-09-17T10:07:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 2 all 8 watchers are set, enabling cache 2021-09-17T10:07:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2021-09-17T10:07:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2021-09-17T10:07:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T10:07:24.369 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2021-09-17T10:07:24.369 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2021-09-17T10:07:24.369 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T10:07:24.369 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2021-09-17T10:07:24.370 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2021-09-17T10:07:24.374 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T10:07:24.375 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2021-09-17T10:07:24.375 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2021-09-17T10:07:24.375 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T10:07:24.375 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2021-09-17T10:07:24.376 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.382+0000 7fcefd695b00 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2021-09-17T10:07:24.376 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.386+0000 7fcefd695b00 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-17T10:07:24.377 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.390+0000 7fce47fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T10:07:24.377 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.390+0000 7fcefd695b00 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T10:07:24.377 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:24.390+0000 7fcefd695b00 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T10:07:28.363 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.380+0000 7fcefd695b00 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-09-17T10:07:28.363 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.380+0000 7fcefd695b00 5 note: GC not initialized 2021-09-17T10:07:28.367 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.384+0000 7fce32ffd700 20 reqs_thread_entry: start 2021-09-17T10:07:28.367 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.384+0000 7fce30ff9700 20 reqs_thread_entry: start 2021-09-17T10:07:28.367 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.384+0000 7fcefd695b00 20 init_complete bucket index max shards: 11 2021-09-17T10:07:28.367 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.384+0000 7fcefd695b00 10 rgw notify: Started notification manager with: 1 workers 2021-09-17T10:07:28.368 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.384+0000 7fce22ffd700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 17 10:07:58 2021 2021-09-17T10:07:28.368 INFO:teuthology.orchestra.run.smithi050.stderr: 2021-09-17T10:07:28.368 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.384+0000 7fce227fc700 20 reqs_thread_entry: start 2021-09-17T10:07:28.392 INFO:teuthology.orchestra.run.smithi050.stdout:[] 2021-09-17T10:07:28.393 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.408+0000 7fcefd695b00 20 remove_watcher() i=0 2021-09-17T10:07:28.393 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.408+0000 7fcefd695b00 2 removed watcher, disabling cache 2021-09-17T10:07:28.393 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.412+0000 7fcefd695b00 20 remove_watcher() i=1 2021-09-17T10:07:28.395 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.412+0000 7fcefd695b00 20 remove_watcher() i=2 2021-09-17T10:07:28.396 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.412+0000 7fcefd695b00 20 remove_watcher() i=3 2021-09-17T10:07:28.398 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.416+0000 7fcefd695b00 20 remove_watcher() i=4 2021-09-17T10:07:28.400 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.416+0000 7fcefd695b00 20 remove_watcher() i=5 2021-09-17T10:07:28.401 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.420+0000 7fcefd695b00 20 remove_watcher() i=6 2021-09-17T10:07:28.403 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.420+0000 7fcefd695b00 20 remove_watcher() i=7 2021-09-17T10:07:28.405 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.420+0000 7fce47fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T10:07:28.413 DEBUG:tasks.util.rgw: json result: [] 2021-09-17T10:07:28.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T10:07:28.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T10:07:28.414 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-17T10:07:28.455 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:28.455 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:28.469 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.484+0000 7f8122588b00 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T10:07:28.469 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.484+0000 7f8122588b00 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T10:07:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.492+0000 7f8069ffb700 20 reqs_thread_entry: start 2021-09-17T10:07:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.492+0000 7f8122588b00 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-17T10:07:28.477 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.492+0000 7f8122588b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.478 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.496+0000 7f8122588b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:28.478 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.496+0000 7f8122588b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.479 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.496+0000 7f8122588b00 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T10:07:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.516+0000 7f8122588b00 20 RGWRados::pool_iterate: got zone_info.94df82a5-cc4b-4217-94b9-aec93a1b010e 2021-09-17T10:07:28.501 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.516+0000 7f8122588b00 20 RGWRados::pool_iterate: got zonegroup_info.c34e6d9b-21b3-4d8d-bda2-2d08ebd7d572 2021-09-17T10:07:28.502 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.516+0000 7f8122588b00 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T10:07:28.502 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.516+0000 7f8122588b00 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T10:07:28.502 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.516+0000 7f8122588b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.520+0000 7f8122588b00 20 rados_obj.operate() r=0 bl.length=873 2021-09-17T10:07:28.503 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.520+0000 7f8122588b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.520+0000 7f8122588b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.520+0000 7f8122588b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.520+0000 7f8122588b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:28.504 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.520+0000 7f8122588b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.505 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.520+0000 7f8122588b00 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T10:07:28.513 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:07:28.513 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "94df82a5-cc4b-4217-94b9-aec93a1b010e", 2021-09-17T10:07:28.514 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "default", 2021-09-17T10:07:28.514 INFO:teuthology.orchestra.run.smithi050.stdout: "domain_root": "default.rgw.meta:root", 2021-09-17T10:07:28.514 INFO:teuthology.orchestra.run.smithi050.stdout: "control_pool": "default.rgw.control", 2021-09-17T10:07:28.514 INFO:teuthology.orchestra.run.smithi050.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-17T10:07:28.515 INFO:teuthology.orchestra.run.smithi050.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-17T10:07:28.515 INFO:teuthology.orchestra.run.smithi050.stdout: "log_pool": "default.rgw.log", 2021-09-17T10:07:28.515 INFO:teuthology.orchestra.run.smithi050.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-17T10:07:28.515 INFO:teuthology.orchestra.run.smithi050.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-17T10:07:28.516 INFO:teuthology.orchestra.run.smithi050.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-17T10:07:28.516 INFO:teuthology.orchestra.run.smithi050.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-17T10:07:28.516 INFO:teuthology.orchestra.run.smithi050.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-17T10:07:28.516 INFO:teuthology.orchestra.run.smithi050.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-17T10:07:28.517 INFO:teuthology.orchestra.run.smithi050.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-17T10:07:28.517 INFO:teuthology.orchestra.run.smithi050.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-17T10:07:28.517 INFO:teuthology.orchestra.run.smithi050.stdout: "otp_pool": "default.rgw.otp", 2021-09-17T10:07:28.517 INFO:teuthology.orchestra.run.smithi050.stdout: "system_key": { 2021-09-17T10:07:28.517 INFO:teuthology.orchestra.run.smithi050.stdout: "access_key": "", 2021-09-17T10:07:28.518 INFO:teuthology.orchestra.run.smithi050.stdout: "secret_key": "" 2021-09-17T10:07:28.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:28.518 INFO:teuthology.orchestra.run.smithi050.stdout: "placement_pools": [ 2021-09-17T10:07:28.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:07:28.519 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "default-placement", 2021-09-17T10:07:28.519 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:07:28.519 INFO:teuthology.orchestra.run.smithi050.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-17T10:07:28.519 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_classes": { 2021-09-17T10:07:28.519 INFO:teuthology.orchestra.run.smithi050.stdout: "STANDARD": { 2021-09-17T10:07:28.520 INFO:teuthology.orchestra.run.smithi050.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-17T10:07:28.520 INFO:teuthology.orchestra.run.smithi050.stdout: "compression_type": "random" 2021-09-17T10:07:28.520 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:28.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:28.521 INFO:teuthology.orchestra.run.smithi050.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-17T10:07:28.521 INFO:teuthology.orchestra.run.smithi050.stdout: "index_type": 0 2021-09-17T10:07:28.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:28.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:28.521 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:07:28.522 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "", 2021-09-17T10:07:28.522 INFO:teuthology.orchestra.run.smithi050.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-17T10:07:28.522 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:07:28.523 DEBUG:tasks.util.rgw: json result: {'id': '94df82a5-cc4b-4217-94b9-aec93a1b010e', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-17T10:07:28.524 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-09-17T10:07:28.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-09-17T10:07:28.525 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-09-17T10:07:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:28.601 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:28.611 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.628+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.614 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.632+0000 7f0848b3eb00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:28.615 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.632+0000 7f0848b3eb00 20 realm 2021-09-17T10:07:28.615 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.632+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.616 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.632+0000 7f0848b3eb00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:28.616 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.632+0000 7f0848b3eb00 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-17T10:07:28.617 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.632+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.617 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.632+0000 7f0848b3eb00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:28.617 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.632+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.617 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.632+0000 7f0848b3eb00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:28.618 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.632+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.618 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.636+0000 7f0848b3eb00 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T10:07:28.619 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.636+0000 7f0848b3eb00 20 searching for the correct realm 2021-09-17T10:07:28.641 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.656+0000 7f0848b3eb00 20 RGWRados::pool_iterate: got zone_info.94df82a5-cc4b-4217-94b9-aec93a1b010e 2021-09-17T10:07:28.641 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 RGWRados::pool_iterate: got zonegroup_info.c34e6d9b-21b3-4d8d-bda2-2d08ebd7d572 2021-09-17T10:07:28.642 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T10:07:28.642 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T10:07:28.642 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.643 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T10:07:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.646 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:28.646 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.646 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:28.646 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.647 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.660+0000 7f0848b3eb00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:28.669 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 RGWRados::pool_iterate: got zone_info.94df82a5-cc4b-4217-94b9-aec93a1b010e 2021-09-17T10:07:28.670 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 RGWRados::pool_iterate: got zonegroup_info.c34e6d9b-21b3-4d8d-bda2-2d08ebd7d572 2021-09-17T10:07:28.670 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T10:07:28.670 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T10:07:28.670 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.671 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:28.671 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.672 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T10:07:28.672 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.672 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:28.673 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.674 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T10:07:28.674 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.688+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.674 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:28.674 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 20 zone default found 2021-09-17T10:07:28.675 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 1 Realm: () 2021-09-17T10:07:28.675 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 1 ZoneGroup: default (c34e6d9b-21b3-4d8d-bda2-2d08ebd7d572) 2021-09-17T10:07:28.675 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 1 Zone: default (94df82a5-cc4b-4217-94b9-aec93a1b010e) 2021-09-17T10:07:28.675 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-17T10:07:28.676 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 20 zonegroup default 2021-09-17T10:07:28.676 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.677 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:28.677 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.677 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:28.678 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.678 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:28.678 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 20 started sync module instance, tier type = 2021-09-17T10:07:28.678 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.692+0000 7f0848b3eb00 20 started zone id=94df82a5-cc4b-4217-94b9-aec93a1b010e (name=default) with tier type = 2021-09-17T10:07:28.694 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.712+0000 7f0848b3eb00 20 add_watcher() i=0 2021-09-17T10:07:28.694 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.712+0000 7f0848b3eb00 20 add_watcher() i=1 2021-09-17T10:07:28.694 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.712+0000 7f0848b3eb00 20 add_watcher() i=2 2021-09-17T10:07:28.695 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.712+0000 7f0848b3eb00 20 add_watcher() i=3 2021-09-17T10:07:28.695 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.712+0000 7f0848b3eb00 20 add_watcher() i=4 2021-09-17T10:07:28.695 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.712+0000 7f0848b3eb00 20 add_watcher() i=5 2021-09-17T10:07:28.696 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.712+0000 7f0848b3eb00 20 add_watcher() i=6 2021-09-17T10:07:28.696 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.712+0000 7f0848b3eb00 20 add_watcher() i=7 2021-09-17T10:07:28.696 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.712+0000 7f0848b3eb00 2 all 8 watchers are set, enabling cache 2021-09-17T10:07:28.698 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.716+0000 7f0817fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T10:07:28.698 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.716+0000 7f0848b3eb00 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T10:07:28.698 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.716+0000 7f0848b3eb00 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T10:07:28.704 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.720+0000 7f0848b3eb00 5 note: GC not initialized 2021-09-17T10:07:28.705 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.724+0000 7f0786ffd700 20 reqs_thread_entry: start 2021-09-17T10:07:28.706 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.724+0000 7f0784ff9700 20 reqs_thread_entry: start 2021-09-17T10:07:28.706 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.724+0000 7f0848b3eb00 20 init_complete bucket index max shards: 11 2021-09-17T10:07:28.707 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.724+0000 7f0848b3eb00 10 rgw notify: Started notification manager with: 1 workers 2021-09-17T10:07:28.707 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.724+0000 7f077affd700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 17 10:07:58 2021 2021-09-17T10:07:28.707 INFO:teuthology.orchestra.run.smithi050.stderr: 2021-09-17T10:07:28.707 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.724+0000 7f077a7fc700 20 reqs_thread_entry: start 2021-09-17T10:07:28.731 INFO:teuthology.orchestra.run.smithi050.stdout:[] 2021-09-17T10:07:28.731 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.748+0000 7f0848b3eb00 20 remove_watcher() i=0 2021-09-17T10:07:28.732 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.748+0000 7f0848b3eb00 2 removed watcher, disabling cache 2021-09-17T10:07:28.733 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.748+0000 7f0848b3eb00 20 remove_watcher() i=1 2021-09-17T10:07:28.735 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.752+0000 7f0848b3eb00 20 remove_watcher() i=2 2021-09-17T10:07:28.738 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.756+0000 7f0848b3eb00 20 remove_watcher() i=3 2021-09-17T10:07:28.740 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.756+0000 7f0848b3eb00 20 remove_watcher() i=4 2021-09-17T10:07:28.742 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.760+0000 7f0848b3eb00 20 remove_watcher() i=5 2021-09-17T10:07:28.745 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.760+0000 7f0848b3eb00 20 remove_watcher() i=6 2021-09-17T10:07:28.747 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.764+0000 7f0848b3eb00 20 remove_watcher() i=7 2021-09-17T10:07:28.749 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.764+0000 7f0817fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-17T10:07:28.758 DEBUG:tasks.util.rgw: json result: [] 2021-09-17T10:07:28.759 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T10:07:28.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-17T10:07:28.759 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-17T10:07:28.801 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:28.802 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.828+0000 7f3aae3c3b00 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-17T10:07:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.828+0000 7f3aae3c3b00 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-17T10:07:28.821 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.836+0000 7f3a057fa700 20 reqs_thread_entry: start 2021-09-17T10:07:28.821 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.836+0000 7f3aae3c3b00 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-17T10:07:28.821 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.836+0000 7f3aae3c3b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.823 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.840+0000 7f3aae3c3b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:28.823 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.840+0000 7f3aae3c3b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.824 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.840+0000 7f3aae3c3b00 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T10:07:28.850 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.864+0000 7f3aae3c3b00 20 RGWRados::pool_iterate: got zone_info.94df82a5-cc4b-4217-94b9-aec93a1b010e 2021-09-17T10:07:28.850 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.864+0000 7f3aae3c3b00 20 RGWRados::pool_iterate: got zonegroup_info.c34e6d9b-21b3-4d8d-bda2-2d08ebd7d572 2021-09-17T10:07:28.850 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.864+0000 7f3aae3c3b00 20 RGWRados::pool_iterate: got zone_names.default 2021-09-17T10:07:28.851 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.864+0000 7f3aae3c3b00 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-17T10:07:28.851 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.864+0000 7f3aae3c3b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.851 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.868+0000 7f3aae3c3b00 20 rados_obj.operate() r=0 bl.length=889 2021-09-17T10:07:28.851 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.868+0000 7f3aae3c3b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.852 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.868+0000 7f3aae3c3b00 20 rados_obj.operate() r=-2 bl.length=0 2021-09-17T10:07:28.852 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.868+0000 7f3aae3c3b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.853 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.868+0000 7f3aae3c3b00 20 rados_obj.operate() r=0 bl.length=46 2021-09-17T10:07:28.853 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.868+0000 7f3aae3c3b00 20 rados->read ofs=0 len=0 2021-09-17T10:07:28.853 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:28.868+0000 7f3aae3c3b00 20 rados_obj.operate() r=0 bl.length=358 2021-09-17T10:07:28.862 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:07:28.862 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "94df82a5-cc4b-4217-94b9-aec93a1b010e", 2021-09-17T10:07:28.862 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "default", 2021-09-17T10:07:28.862 INFO:teuthology.orchestra.run.smithi050.stdout: "domain_root": "default.rgw.meta:root", 2021-09-17T10:07:28.863 INFO:teuthology.orchestra.run.smithi050.stdout: "control_pool": "default.rgw.control", 2021-09-17T10:07:28.863 INFO:teuthology.orchestra.run.smithi050.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-17T10:07:28.863 INFO:teuthology.orchestra.run.smithi050.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-17T10:07:28.863 INFO:teuthology.orchestra.run.smithi050.stdout: "log_pool": "default.rgw.log", 2021-09-17T10:07:28.863 INFO:teuthology.orchestra.run.smithi050.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-17T10:07:28.864 INFO:teuthology.orchestra.run.smithi050.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-17T10:07:28.864 INFO:teuthology.orchestra.run.smithi050.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-17T10:07:28.864 INFO:teuthology.orchestra.run.smithi050.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-17T10:07:28.864 INFO:teuthology.orchestra.run.smithi050.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-17T10:07:28.865 INFO:teuthology.orchestra.run.smithi050.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-17T10:07:28.865 INFO:teuthology.orchestra.run.smithi050.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-17T10:07:28.865 INFO:teuthology.orchestra.run.smithi050.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-17T10:07:28.865 INFO:teuthology.orchestra.run.smithi050.stdout: "otp_pool": "default.rgw.otp", 2021-09-17T10:07:28.865 INFO:teuthology.orchestra.run.smithi050.stdout: "system_key": { 2021-09-17T10:07:28.866 INFO:teuthology.orchestra.run.smithi050.stdout: "access_key": "", 2021-09-17T10:07:28.866 INFO:teuthology.orchestra.run.smithi050.stdout: "secret_key": "" 2021-09-17T10:07:28.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:28.866 INFO:teuthology.orchestra.run.smithi050.stdout: "placement_pools": [ 2021-09-17T10:07:28.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:07:28.867 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "default-placement", 2021-09-17T10:07:28.867 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:07:28.867 INFO:teuthology.orchestra.run.smithi050.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-17T10:07:28.867 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_classes": { 2021-09-17T10:07:28.868 INFO:teuthology.orchestra.run.smithi050.stdout: "STANDARD": { 2021-09-17T10:07:28.868 INFO:teuthology.orchestra.run.smithi050.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-17T10:07:28.868 INFO:teuthology.orchestra.run.smithi050.stdout: "compression_type": "random" 2021-09-17T10:07:28.869 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:28.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:28.869 INFO:teuthology.orchestra.run.smithi050.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-17T10:07:28.869 INFO:teuthology.orchestra.run.smithi050.stdout: "index_type": 0 2021-09-17T10:07:28.869 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:28.870 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:28.870 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:07:28.870 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "", 2021-09-17T10:07:28.870 INFO:teuthology.orchestra.run.smithi050.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-17T10:07:28.871 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:07:28.872 DEBUG:tasks.util.rgw: json result: {'id': '94df82a5-cc4b-4217-94b9-aec93a1b010e', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-17T10:07:28.872 INFO:tasks.rgw:Starting rgw... 2021-09-17T10:07:28.872 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-09-17T10:07:28.873 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-17T10:07:28.873 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-09-17T10:07:28.874 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-09-17T10:07:28.877 INFO:tasks.rgw.c1.client.0:Started 2021-09-17T10:07:28.877 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-09-17T10:07:28.877 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-17T10:07:28.878 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-09-17T10:07:28.878 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-09-17T10:07:28.880 INFO:tasks.rgw.c1.client.1:Started 2021-09-17T10:07:28.881 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-09-17T10:07:28.881 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-17T10:07:28.881 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-09-17T10:07:28.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-09-17T10:07:28.884 INFO:tasks.rgw.c2.client.0:Started 2021-09-17T10:07:28.885 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-09-17T10:07:28.885 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-17T10:07:28.885 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-09-17T10:07:28.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-09-17T10:07:28.888 INFO:tasks.rgw.c2.client.1:Started 2021-09-17T10:07:28.889 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi018.front.sepia.ceph.com:8000/ 2021-09-17T10:07:28.889 DEBUG:teuthology.orchestra.run.smithi018:> curl http://smithi018.front.sepia.ceph.com:8000/ 2021-09-17T10:07:28.917 INFO:teuthology.orchestra.run.smithi018.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:07:28.918 INFO:teuthology.orchestra.run.smithi018.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:07:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-17T10:07:28.921 INFO:teuthology.orchestra.run.smithi018.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2021-09-17T10:07:28.921 INFO:teuthology.orchestra.run.smithi018.stderr:curl: (7) Failed to connect to smithi018.front.sepia.ceph.com port 8000: Connection refused 2021-09-17T10:07:29.922 DEBUG:teuthology.orchestra.run.smithi018:> curl http://smithi018.front.sepia.ceph.com:8000/ 2021-09-17T10:07:29.941 INFO:teuthology.orchestra.run.smithi018.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:07:29.941 INFO:teuthology.orchestra.run.smithi018.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:07:29.945 INFO:teuthology.orchestra.run.smithi018.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2021-09-17T10:07:29.945 INFO:teuthology.orchestra.run.smithi018.stdout:anonymous 2021-09-17T10:07:29.946 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi018.front.sepia.ceph.com:8001/ 2021-09-17T10:07:29.946 DEBUG:teuthology.orchestra.run.smithi018:> curl http://smithi018.front.sepia.ceph.com:8001/ 2021-09-17T10:07:29.960 INFO:teuthology.orchestra.run.smithi018.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:07:29.960 INFO:teuthology.orchestra.run.smithi018.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:07:29.964 INFO:teuthology.orchestra.run.smithi018.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2021-09-17T10:07:29.965 INFO:teuthology.orchestra.run.smithi018.stdout:anonymous 2021-09-17T10:07:29.965 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi050.front.sepia.ceph.com:8000/ 2021-09-17T10:07:29.966 DEBUG:teuthology.orchestra.run.smithi050:> curl http://smithi050.front.sepia.ceph.com:8000/ 2021-09-17T10:07:29.996 INFO:teuthology.orchestra.run.smithi050.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:07:29.996 INFO:teuthology.orchestra.run.smithi050.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:07:30.000 INFO:teuthology.orchestra.run.smithi050.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2021-09-17T10:07:30.001 INFO:teuthology.orchestra.run.smithi050.stdout:anonymous 2021-09-17T10:07:30.001 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi050.front.sepia.ceph.com:8001/ 2021-09-17T10:07:30.002 DEBUG:teuthology.orchestra.run.smithi050:> curl http://smithi050.front.sepia.ceph.com:8001/ 2021-09-17T10:07:30.056 INFO:teuthology.orchestra.run.smithi050.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:07:30.056 INFO:teuthology.orchestra.run.smithi050.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:07:30.060 INFO:teuthology.orchestra.run.smithi050.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2021-09-17T10:07:30.062 INFO:teuthology.orchestra.run.smithi050.stdout:anonymous 2021-09-17T10:07:30.062 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-09-17T10:07:30.388 INFO:tasks.rgw_multisite:creating realm.. 2021-09-17T10:07:30.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:30.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:30.390 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:07:30.427 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:30.427 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:30.501 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:07:30.501 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:07:30.502 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-realm", 2021-09-17T10:07:30.502 INFO:teuthology.orchestra.run.smithi018.stdout: "current_period": "20a9da20-85d6-489f-bcfe-ebc80031614a", 2021-09-17T10:07:30.502 INFO:teuthology.orchestra.run.smithi018.stdout: "epoch": 1 2021-09-17T10:07:30.502 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:07:30.504 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-09-17T10:07:30.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:30.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:30.505 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:07:30.586 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:30.586 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:30.611 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:30.628+0000 7f78b681ab00 -1 old format 2021-09-17T10:07:30.626 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:07:30.626 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "test-zonegroup", 2021-09-17T10:07:30.626 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zonegroup", 2021-09-17T10:07:30.627 INFO:teuthology.orchestra.run.smithi018.stdout: "api_name": "", 2021-09-17T10:07:30.627 INFO:teuthology.orchestra.run.smithi018.stdout: "is_master": "true", 2021-09-17T10:07:30.627 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:07:30.627 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:07:30.628 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:07:30.628 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames": [], 2021-09-17T10:07:30.628 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames_s3website": [], 2021-09-17T10:07:30.628 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zone": "", 2021-09-17T10:07:30.629 INFO:teuthology.orchestra.run.smithi018.stdout: "zones": [], 2021-09-17T10:07:30.629 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_targets": [], 2021-09-17T10:07:30.629 INFO:teuthology.orchestra.run.smithi018.stdout: "default_placement": "", 2021-09-17T10:07:30.629 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:07:30.630 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_policy": { 2021-09-17T10:07:30.630 INFO:teuthology.orchestra.run.smithi018.stdout: "groups": [] 2021-09-17T10:07:30.630 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:30.630 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:07:30.631 INFO:tasks.rgw_multisite:creating master zone.. 2021-09-17T10:07:30.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--endpoints', 'http://smithi018.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:30.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--endpoints', 'http://smithi018.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:30.633 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key H92RM4Y9KBWIE4BC --secret RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK --endpoints http://smithi018.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:07:30.681 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:30.682 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:30.748 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:07:30.748 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:07:30.748 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone1", 2021-09-17T10:07:30.749 INFO:teuthology.orchestra.run.smithi018.stdout: "domain_root": "test-zone1.rgw.meta:root", 2021-09-17T10:07:30.749 INFO:teuthology.orchestra.run.smithi018.stdout: "control_pool": "test-zone1.rgw.control", 2021-09-17T10:07:30.749 INFO:teuthology.orchestra.run.smithi018.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2021-09-17T10:07:30.749 INFO:teuthology.orchestra.run.smithi018.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2021-09-17T10:07:30.750 INFO:teuthology.orchestra.run.smithi018.stdout: "log_pool": "test-zone1.rgw.log", 2021-09-17T10:07:30.750 INFO:teuthology.orchestra.run.smithi018.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2021-09-17T10:07:30.750 INFO:teuthology.orchestra.run.smithi018.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2021-09-17T10:07:30.750 INFO:teuthology.orchestra.run.smithi018.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2021-09-17T10:07:30.751 INFO:teuthology.orchestra.run.smithi018.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2021-09-17T10:07:30.751 INFO:teuthology.orchestra.run.smithi018.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2021-09-17T10:07:30.751 INFO:teuthology.orchestra.run.smithi018.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2021-09-17T10:07:30.751 INFO:teuthology.orchestra.run.smithi018.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2021-09-17T10:07:30.752 INFO:teuthology.orchestra.run.smithi018.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2021-09-17T10:07:30.752 INFO:teuthology.orchestra.run.smithi018.stdout: "otp_pool": "test-zone1.rgw.otp", 2021-09-17T10:07:30.752 INFO:teuthology.orchestra.run.smithi018.stdout: "system_key": { 2021-09-17T10:07:30.752 INFO:teuthology.orchestra.run.smithi018.stdout: "access_key": "H92RM4Y9KBWIE4BC", 2021-09-17T10:07:30.753 INFO:teuthology.orchestra.run.smithi018.stdout: "secret_key": "RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK" 2021-09-17T10:07:30.753 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:30.753 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_pools": [ 2021-09-17T10:07:30.753 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:07:30.753 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "default-placement", 2021-09-17T10:07:30.754 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:07:30.754 INFO:teuthology.orchestra.run.smithi018.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2021-09-17T10:07:30.754 INFO:teuthology.orchestra.run.smithi018.stdout: "storage_classes": { 2021-09-17T10:07:30.754 INFO:teuthology.orchestra.run.smithi018.stdout: "STANDARD": { 2021-09-17T10:07:30.755 INFO:teuthology.orchestra.run.smithi018.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2021-09-17T10:07:30.755 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:30.755 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:30.755 INFO:teuthology.orchestra.run.smithi018.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2021-09-17T10:07:30.756 INFO:teuthology.orchestra.run.smithi018.stdout: "index_type": 0 2021-09-17T10:07:30.756 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:30.756 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:30.756 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:07:30.756 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:07:30.757 INFO:teuthology.orchestra.run.smithi018.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2021-09-17T10:07:30.757 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:07:30.758 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-09-17T10:07:32.027 INFO:teuthology.orchestra.run.smithi018.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-09-17T10:07:32.049 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-09-17T10:07:33.055 INFO:teuthology.orchestra.run.smithi018.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-09-17T10:07:33.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:33.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:33.073 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:07:33.108 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:33.108 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:33.159 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:07:33.159 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:07:33.160 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone1", 2021-09-17T10:07:33.160 INFO:teuthology.orchestra.run.smithi018.stdout: "domain_root": "test-zone1.rgw.meta:root", 2021-09-17T10:07:33.160 INFO:teuthology.orchestra.run.smithi018.stdout: "control_pool": "test-zone1.rgw.control", 2021-09-17T10:07:33.160 INFO:teuthology.orchestra.run.smithi018.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2021-09-17T10:07:33.161 INFO:teuthology.orchestra.run.smithi018.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2021-09-17T10:07:33.161 INFO:teuthology.orchestra.run.smithi018.stdout: "log_pool": "test-zone1.rgw.log", 2021-09-17T10:07:33.161 INFO:teuthology.orchestra.run.smithi018.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2021-09-17T10:07:33.161 INFO:teuthology.orchestra.run.smithi018.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2021-09-17T10:07:33.162 INFO:teuthology.orchestra.run.smithi018.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2021-09-17T10:07:33.162 INFO:teuthology.orchestra.run.smithi018.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2021-09-17T10:07:33.162 INFO:teuthology.orchestra.run.smithi018.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2021-09-17T10:07:33.162 INFO:teuthology.orchestra.run.smithi018.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2021-09-17T10:07:33.163 INFO:teuthology.orchestra.run.smithi018.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2021-09-17T10:07:33.163 INFO:teuthology.orchestra.run.smithi018.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2021-09-17T10:07:33.163 INFO:teuthology.orchestra.run.smithi018.stdout: "otp_pool": "test-zone1.rgw.otp", 2021-09-17T10:07:33.163 INFO:teuthology.orchestra.run.smithi018.stdout: "system_key": { 2021-09-17T10:07:33.163 INFO:teuthology.orchestra.run.smithi018.stdout: "access_key": "H92RM4Y9KBWIE4BC", 2021-09-17T10:07:33.164 INFO:teuthology.orchestra.run.smithi018.stdout: "secret_key": "RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK" 2021-09-17T10:07:33.164 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:33.164 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_pools": [ 2021-09-17T10:07:33.164 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:07:33.165 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "default-placement", 2021-09-17T10:07:33.165 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:07:33.165 INFO:teuthology.orchestra.run.smithi018.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2021-09-17T10:07:33.165 INFO:teuthology.orchestra.run.smithi018.stdout: "storage_classes": { 2021-09-17T10:07:33.165 INFO:teuthology.orchestra.run.smithi018.stdout: "STANDARD": { 2021-09-17T10:07:33.166 INFO:teuthology.orchestra.run.smithi018.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2021-09-17T10:07:33.166 INFO:teuthology.orchestra.run.smithi018.stdout: "compression_type": "random" 2021-09-17T10:07:33.166 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:33.166 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:33.167 INFO:teuthology.orchestra.run.smithi018.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2021-09-17T10:07:33.167 INFO:teuthology.orchestra.run.smithi018.stdout: "index_type": 0 2021-09-17T10:07:33.167 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:33.167 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:33.168 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:07:33.168 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "", 2021-09-17T10:07:33.168 INFO:teuthology.orchestra.run.smithi018.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2021-09-17T10:07:33.168 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:07:33.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:33.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:33.170 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:07:33.210 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:33.210 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:33.230 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:07:33.244+0000 7f09e3a7fb00 0 period (20a9da20-85d6-489f-bcfe-ebc80031614a does not have zone 9854c5a3-6cc0-45a7-b530-3af2b1c733d5 configured 2021-09-17T10:07:41.192 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:07:41.192 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:07:41.192 INFO:teuthology.orchestra.run.smithi018.stdout: "epoch": 1, 2021-09-17T10:07:41.192 INFO:teuthology.orchestra.run.smithi018.stdout: "predecessor_uuid": "20a9da20-85d6-489f-bcfe-ebc80031614a", 2021-09-17T10:07:41.193 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": [], 2021-09-17T10:07:41.193 INFO:teuthology.orchestra.run.smithi018.stdout: "period_map": { 2021-09-17T10:07:41.193 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:07:41.193 INFO:teuthology.orchestra.run.smithi018.stdout: "zonegroups": [ 2021-09-17T10:07:41.194 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:07:41.194 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "test-zonegroup", 2021-09-17T10:07:41.194 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zonegroup", 2021-09-17T10:07:41.194 INFO:teuthology.orchestra.run.smithi018.stdout: "api_name": "", 2021-09-17T10:07:41.195 INFO:teuthology.orchestra.run.smithi018.stdout: "is_master": "true", 2021-09-17T10:07:41.195 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:07:41.195 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:07:41.195 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:07:41.196 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames": [], 2021-09-17T10:07:41.196 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames_s3website": [], 2021-09-17T10:07:41.196 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:07:41.196 INFO:teuthology.orchestra.run.smithi018.stdout: "zones": [ 2021-09-17T10:07:41.196 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:07:41.197 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:07:41.197 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone1", 2021-09-17T10:07:41.197 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:07:41.197 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:07:41.198 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:07:41.198 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:07:41.198 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "false", 2021-09-17T10:07:41.198 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:07:41.198 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:07:41.199 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "", 2021-09-17T10:07:41.199 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "true", 2021-09-17T10:07:41.199 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [], 2021-09-17T10:07:41.199 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:07:41.200 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:41.200 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:07:41.200 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_targets": [ 2021-09-17T10:07:41.200 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:07:41.201 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "default-placement", 2021-09-17T10:07:41.201 INFO:teuthology.orchestra.run.smithi018.stdout: "tags": [], 2021-09-17T10:07:41.201 INFO:teuthology.orchestra.run.smithi018.stdout: "storage_classes": [ 2021-09-17T10:07:41.201 INFO:teuthology.orchestra.run.smithi018.stdout: "STANDARD" 2021-09-17T10:07:41.201 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:07:41.202 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:41.202 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:07:41.202 INFO:teuthology.orchestra.run.smithi018.stdout: "default_placement": "default-placement", 2021-09-17T10:07:41.202 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:07:41.203 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_policy": { 2021-09-17T10:07:41.203 INFO:teuthology.orchestra.run.smithi018.stdout: "groups": [] 2021-09-17T10:07:41.203 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:41.203 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:41.203 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:07:41.204 INFO:teuthology.orchestra.run.smithi018.stdout: "short_zone_ids": [ 2021-09-17T10:07:41.204 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:07:41.204 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:07:41.204 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 664423460 2021-09-17T10:07:41.204 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:41.205 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:07:41.205 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:41.205 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T10:07:41.205 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:07:41.206 INFO:teuthology.orchestra.run.smithi018.stdout: "period_config": { 2021-09-17T10:07:41.206 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_quota": { 2021-09-17T10:07:41.206 INFO:teuthology.orchestra.run.smithi018.stdout: "enabled": false, 2021-09-17T10:07:41.206 INFO:teuthology.orchestra.run.smithi018.stdout: "check_on_raw": false, 2021-09-17T10:07:41.206 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size": -1, 2021-09-17T10:07:41.207 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size_kb": 0, 2021-09-17T10:07:41.207 INFO:teuthology.orchestra.run.smithi018.stdout: "max_objects": -1 2021-09-17T10:07:41.207 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:41.207 INFO:teuthology.orchestra.run.smithi018.stdout: "user_quota": { 2021-09-17T10:07:41.208 INFO:teuthology.orchestra.run.smithi018.stdout: "enabled": false, 2021-09-17T10:07:41.208 INFO:teuthology.orchestra.run.smithi018.stdout: "check_on_raw": false, 2021-09-17T10:07:41.208 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size": -1, 2021-09-17T10:07:41.208 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size_kb": 0, 2021-09-17T10:07:41.208 INFO:teuthology.orchestra.run.smithi018.stdout: "max_objects": -1 2021-09-17T10:07:41.209 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:41.209 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:41.209 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:07:41.209 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_name": "test-realm", 2021-09-17T10:07:41.209 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:07:41.210 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:07:41.211 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-09-17T10:07:41.211 INFO:teuthology.orchestra.run:waiting for 300 2021-09-17T10:07:41.256 INFO:tasks.rgw.c1.client.0.smithi018.stdout:2021-09-17T10:07:41.272+0000 7f1996ffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 17984) UID: 0 2021-09-17T10:07:41.256 INFO:tasks.rgw.c1.client.0.smithi018.stdout:2021-09-17T10:07:41.272+0000 7f19a7a8fb40 -1 shutting down 2021-09-17T10:07:42.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-09-17T10:07:47.313 INFO:tasks.rgw.c1.client.0:Stopped 2021-09-17T10:07:47.314 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-09-17T10:07:47.314 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-09-17T10:07:47.317 INFO:tasks.rgw.c1.client.0:Started 2021-09-17T10:07:47.318 DEBUG:teuthology.orchestra.run.smithi018:> curl http://smithi018.front.sepia.ceph.com:8000 2021-09-17T10:07:47.335 INFO:teuthology.orchestra.run.smithi018.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:07:47.335 INFO:teuthology.orchestra.run.smithi018.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:07:47.336 INFO:teuthology.orchestra.run.smithi018.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2021-09-17T10:07:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-17T10:07:47.338 INFO:teuthology.orchestra.run.smithi018.stderr:curl: (7) Failed to connect to smithi018.front.sepia.ceph.com port 8000: Connection refused 2021-09-17T10:07:48.340 DEBUG:teuthology.orchestra.run.smithi018:> curl http://smithi018.front.sepia.ceph.com:8000 2021-09-17T10:07:48.355 INFO:teuthology.orchestra.run.smithi018.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:07:48.355 INFO:teuthology.orchestra.run.smithi018.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:07:48.358 INFO:teuthology.orchestra.run.smithi018.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2021-09-17T10:07:48.359 INFO:teuthology.orchestra.run.smithi018.stdout:anonymous 2021-09-17T10:07:48.360 INFO:tasks.rgw_multisite:creating admin user.. 2021-09-17T10:07:48.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:48.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:48.361 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key H92RM4Y9KBWIE4BC --secret RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:07:48.408 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:48.409 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:48.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-09-17T10:07:48.498 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:07:48.499 INFO:teuthology.orchestra.run.smithi018.stdout: "user_id": "realm-admin", 2021-09-17T10:07:48.499 INFO:teuthology.orchestra.run.smithi018.stdout: "display_name": "Realm Admin", 2021-09-17T10:07:48.499 INFO:teuthology.orchestra.run.smithi018.stdout: "email": "", 2021-09-17T10:07:48.499 INFO:teuthology.orchestra.run.smithi018.stdout: "suspended": 0, 2021-09-17T10:07:48.500 INFO:teuthology.orchestra.run.smithi018.stdout: "max_buckets": 1000, 2021-09-17T10:07:48.500 INFO:teuthology.orchestra.run.smithi018.stdout: "subusers": [], 2021-09-17T10:07:48.500 INFO:teuthology.orchestra.run.smithi018.stdout: "keys": [ 2021-09-17T10:07:48.500 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:07:48.500 INFO:teuthology.orchestra.run.smithi018.stdout: "user": "realm-admin", 2021-09-17T10:07:48.501 INFO:teuthology.orchestra.run.smithi018.stdout: "access_key": "H92RM4Y9KBWIE4BC", 2021-09-17T10:07:48.501 INFO:teuthology.orchestra.run.smithi018.stdout: "secret_key": "RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK" 2021-09-17T10:07:48.501 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:07:48.501 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:07:48.502 INFO:teuthology.orchestra.run.smithi018.stdout: "swift_keys": [], 2021-09-17T10:07:48.502 INFO:teuthology.orchestra.run.smithi018.stdout: "caps": [], 2021-09-17T10:07:48.502 INFO:teuthology.orchestra.run.smithi018.stdout: "op_mask": "read, write, delete", 2021-09-17T10:07:48.502 INFO:teuthology.orchestra.run.smithi018.stdout: "system": "true", 2021-09-17T10:07:48.502 INFO:teuthology.orchestra.run.smithi018.stdout: "default_placement": "", 2021-09-17T10:07:48.503 INFO:teuthology.orchestra.run.smithi018.stdout: "default_storage_class": "", 2021-09-17T10:07:48.503 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_tags": [], 2021-09-17T10:07:48.503 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_quota": { 2021-09-17T10:07:48.503 INFO:teuthology.orchestra.run.smithi018.stdout: "enabled": false, 2021-09-17T10:07:48.504 INFO:teuthology.orchestra.run.smithi018.stdout: "check_on_raw": false, 2021-09-17T10:07:48.504 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size": -1, 2021-09-17T10:07:48.504 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size_kb": 0, 2021-09-17T10:07:48.504 INFO:teuthology.orchestra.run.smithi018.stdout: "max_objects": -1 2021-09-17T10:07:48.504 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:48.505 INFO:teuthology.orchestra.run.smithi018.stdout: "user_quota": { 2021-09-17T10:07:48.505 INFO:teuthology.orchestra.run.smithi018.stdout: "enabled": false, 2021-09-17T10:07:48.505 INFO:teuthology.orchestra.run.smithi018.stdout: "check_on_raw": false, 2021-09-17T10:07:48.505 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size": -1, 2021-09-17T10:07:48.506 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size_kb": 0, 2021-09-17T10:07:48.506 INFO:teuthology.orchestra.run.smithi018.stdout: "max_objects": -1 2021-09-17T10:07:48.506 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:07:48.506 INFO:teuthology.orchestra.run.smithi018.stdout: "temp_url_keys": [], 2021-09-17T10:07:48.506 INFO:teuthology.orchestra.run.smithi018.stdout: "type": "rgw", 2021-09-17T10:07:48.507 INFO:teuthology.orchestra.run.smithi018.stdout: "mfa_ids": [] 2021-09-17T10:07:48.507 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:07:48.507 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:07:48.528 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-09-17T10:07:48.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi018.front.sepia.ceph.com:8000', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:48.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi018.front.sepia.ceph.com:8000', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:48.529 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi018.front.sepia.ceph.com:8000 --access-key H92RM4Y9KBWIE4BC --secret RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T10:07:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:48.574 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:48.664 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:07:48.665 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:07:48.665 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-realm", 2021-09-17T10:07:48.665 INFO:teuthology.orchestra.run.smithi050.stdout: "current_period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:07:48.665 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 2 2021-09-17T10:07:48.666 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:07:48.667 INFO:tasks.rgw_multisite:creating zone.. 2021-09-17T10:07:48.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--endpoints', 'http://smithi050.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:48.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--endpoints', 'http://smithi050.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:48.668 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key H92RM4Y9KBWIE4BC --secret RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK --endpoints http://smithi050.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T10:07:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:48.772 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:48.786+0000 7f1f94093b00 0 failed reading obj info from .rgw.root:zone_info.9854c5a3-6cc0-45a7-b530-3af2b1c733d5: (2) No such file or directory 2021-09-17T10:07:48.773 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:48.786+0000 7f1f94093b00 0 WARNING: could not read zone params for zone id=9854c5a3-6cc0-45a7-b530-3af2b1c733d5 name=test-zone1 2021-09-17T10:07:48.783 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:07:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:07:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone2", 2021-09-17T10:07:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-17T10:07:48.784 INFO:teuthology.orchestra.run.smithi050.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-17T10:07:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-17T10:07:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-17T10:07:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-17T10:07:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-17T10:07:48.785 INFO:teuthology.orchestra.run.smithi050.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-17T10:07:48.786 INFO:teuthology.orchestra.run.smithi050.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-17T10:07:48.786 INFO:teuthology.orchestra.run.smithi050.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-17T10:07:48.786 INFO:teuthology.orchestra.run.smithi050.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-17T10:07:48.786 INFO:teuthology.orchestra.run.smithi050.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-17T10:07:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-17T10:07:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-17T10:07:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-17T10:07:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "system_key": { 2021-09-17T10:07:48.787 INFO:teuthology.orchestra.run.smithi050.stdout: "access_key": "H92RM4Y9KBWIE4BC", 2021-09-17T10:07:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: "secret_key": "RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK" 2021-09-17T10:07:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: "placement_pools": [ 2021-09-17T10:07:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:07:48.788 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "default-placement", 2021-09-17T10:07:48.789 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:07:48.789 INFO:teuthology.orchestra.run.smithi050.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-17T10:07:48.789 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_classes": { 2021-09-17T10:07:48.789 INFO:teuthology.orchestra.run.smithi050.stdout: "STANDARD": { 2021-09-17T10:07:48.789 INFO:teuthology.orchestra.run.smithi050.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2021-09-17T10:07:48.790 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:48.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:48.790 INFO:teuthology.orchestra.run.smithi050.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-17T10:07:48.790 INFO:teuthology.orchestra.run.smithi050.stdout: "index_type": 0 2021-09-17T10:07:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:07:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:07:48.791 INFO:teuthology.orchestra.run.smithi050.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-17T10:07:48.792 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:07:48.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-09-17T10:07:49.443 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-09-17T10:07:49.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-09-17T10:07:50.447 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-09-17T10:07:50.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:50.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:50.465 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T10:07:50.499 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:50.500 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:50.551 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:07:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:07:50.552 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone2", 2021-09-17T10:07:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-17T10:07:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-17T10:07:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-17T10:07:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-17T10:07:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-17T10:07:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-17T10:07:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-17T10:07:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-17T10:07:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-17T10:07:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-17T10:07:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-17T10:07:50.555 INFO:teuthology.orchestra.run.smithi050.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-17T10:07:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-17T10:07:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-17T10:07:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: "system_key": { 2021-09-17T10:07:50.556 INFO:teuthology.orchestra.run.smithi050.stdout: "access_key": "H92RM4Y9KBWIE4BC", 2021-09-17T10:07:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: "secret_key": "RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK" 2021-09-17T10:07:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: "placement_pools": [ 2021-09-17T10:07:50.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:07:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "default-placement", 2021-09-17T10:07:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:07:50.558 INFO:teuthology.orchestra.run.smithi050.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-17T10:07:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_classes": { 2021-09-17T10:07:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: "STANDARD": { 2021-09-17T10:07:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2021-09-17T10:07:50.559 INFO:teuthology.orchestra.run.smithi050.stdout: "compression_type": "random" 2021-09-17T10:07:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-17T10:07:50.560 INFO:teuthology.orchestra.run.smithi050.stdout: "index_type": 0 2021-09-17T10:07:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:07:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "", 2021-09-17T10:07:50.561 INFO:teuthology.orchestra.run.smithi050.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-17T10:07:50.562 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:07:50.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:50.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:07:50.564 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T10:07:50.603 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:07:50.603 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:07:50.622 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:07:50.638+0000 7f82f17a0b00 0 period (3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac does not have zone 304080b6-3b8b-4555-8935-e35e25942a93 configured 2021-09-17T10:07:58.533 INFO:teuthology.orchestra.run.smithi050.stderr:Sending period to new master zone 9854c5a3-6cc0-45a7-b530-3af2b1c733d5 2021-09-17T10:07:58.615 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:07:58.615 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:07:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 2, 2021-09-17T10:07:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: "predecessor_uuid": "20a9da20-85d6-489f-bcfe-ebc80031614a", 2021-09-17T10:07:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": [], 2021-09-17T10:07:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: "period_map": { 2021-09-17T10:07:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:07:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: "zonegroups": [ 2021-09-17T10:07:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:07:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "test-zonegroup", 2021-09-17T10:07:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zonegroup", 2021-09-17T10:07:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: "api_name": "", 2021-09-17T10:07:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: "is_master": "true", 2021-09-17T10:07:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:07:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:07:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:07:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "hostnames": [], 2021-09-17T10:07:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "hostnames_s3website": [], 2021-09-17T10:07:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:07:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "zones": [ 2021-09-17T10:07:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:07:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:07:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone2", 2021-09-17T10:07:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:07:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi050.front.sepia.ceph.com:8000" 2021-09-17T10:07:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:07:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: "log_meta": "false", 2021-09-17T10:07:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: "log_data": "true", 2021-09-17T10:07:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:07:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: "read_only": "false", 2021-09-17T10:07:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_type": "", 2021-09-17T10:07:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from_all": "true", 2021-09-17T10:07:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from": [], 2021-09-17T10:07:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_zone": "" 2021-09-17T10:07:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:07:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:07:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone1", 2021-09-17T10:07:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:07:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:07:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:07:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "log_meta": "false", 2021-09-17T10:07:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "log_data": "true", 2021-09-17T10:07:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:07:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "read_only": "false", 2021-09-17T10:07:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_type": "", 2021-09-17T10:07:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from_all": "true", 2021-09-17T10:07:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from": [], 2021-09-17T10:07:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_zone": "" 2021-09-17T10:07:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:07:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "placement_targets": [ 2021-09-17T10:07:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:07:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "default-placement", 2021-09-17T10:07:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2021-09-17T10:07:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_classes": [ 2021-09-17T10:07:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "STANDARD" 2021-09-17T10:07:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:07:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:07:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "default_placement": "default-placement", 2021-09-17T10:07:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:07:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_policy": { 2021-09-17T10:07:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "groups": [] 2021-09-17T10:07:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:07:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "short_zone_ids": [ 2021-09-17T10:07:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:07:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:07:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 1815852058 2021-09-17T10:07:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:07:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:07:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 664423460 2021-09-17T10:07:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:07:58.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:58.632 INFO:teuthology.orchestra.run.smithi050.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T10:07:58.632 INFO:teuthology.orchestra.run.smithi050.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:07:58.632 INFO:teuthology.orchestra.run.smithi050.stdout: "period_config": { 2021-09-17T10:07:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_quota": { 2021-09-17T10:07:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": false, 2021-09-17T10:07:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "check_on_raw": false, 2021-09-17T10:07:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size": -1, 2021-09-17T10:07:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size_kb": 0, 2021-09-17T10:07:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: "max_objects": -1 2021-09-17T10:07:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: "user_quota": { 2021-09-17T10:07:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": false, 2021-09-17T10:07:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "check_on_raw": false, 2021-09-17T10:07:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size": -1, 2021-09-17T10:07:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size_kb": 0, 2021-09-17T10:07:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "max_objects": -1 2021-09-17T10:07:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:07:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:07:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:07:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_name": "test-realm", 2021-09-17T10:07:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:07:58.637 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:07:58.638 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-09-17T10:07:58.638 INFO:teuthology.orchestra.run:waiting for 300 2021-09-17T10:07:58.683 INFO:tasks.rgw.c2.client.0.smithi050.stdout:2021-09-17T10:07:58.698+0000 7f373effd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 18339) UID: 0 2021-09-17T10:07:58.683 INFO:tasks.rgw.c2.client.0.smithi050.stdout:2021-09-17T10:07:58.698+0000 7f374f88bb40 -1 shutting down 2021-09-17T10:08:03.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-09-17T10:08:04.740 INFO:tasks.rgw.c2.client.0:Stopped 2021-09-17T10:08:04.741 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-09-17T10:08:04.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-09-17T10:08:04.744 INFO:tasks.rgw.c2.client.0:Started 2021-09-17T10:08:04.745 DEBUG:teuthology.orchestra.run.smithi050:> curl http://smithi050.front.sepia.ceph.com:8000 2021-09-17T10:08:04.762 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-17T10:08:04.763 INFO:teuthology.orchestra.run.smithi050.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:08:04.764 INFO:teuthology.orchestra.run.smithi050.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:08:04.764 INFO:teuthology.orchestra.run.smithi050.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2021-09-17T10:08:04.764 INFO:teuthology.orchestra.run.smithi050.stderr:curl: (7) Failed to connect to smithi050.front.sepia.ceph.com port 8000: Connection refused 2021-09-17T10:08:05.765 DEBUG:teuthology.orchestra.run.smithi050:> curl http://smithi050.front.sepia.ceph.com:8000 2021-09-17T10:08:05.784 INFO:teuthology.orchestra.run.smithi050.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:08:05.784 INFO:teuthology.orchestra.run.smithi050.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:08:05.787 INFO:teuthology.orchestra.run.smithi050.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2021-09-17T10:08:05.788 INFO:teuthology.orchestra.run.smithi050.stdout:anonymous 2021-09-17T10:08:05.789 INFO:tasks.rgw_multisite:creating zone.. 2021-09-17T10:08:05.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--endpoints', 'http://smithi018.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:05.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--endpoints', 'http://smithi018.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:05.790 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key H92RM4Y9KBWIE4BC --secret RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK --endpoints http://smithi018.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:08:05.837 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:05.838 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:05.897 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:08:05.912+0000 7f445f70db00 0 failed reading obj info from .rgw.root:zone_info.304080b6-3b8b-4555-8935-e35e25942a93: (2) No such file or directory 2021-09-17T10:08:05.897 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:08:05.912+0000 7f445f70db00 0 WARNING: could not read zone params for zone id=304080b6-3b8b-4555-8935-e35e25942a93 name=test-zone2 2021-09-17T10:08:05.907 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:08:05.908 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "d9facd38-bb4e-4ea2-9817-2655cc5c6379", 2021-09-17T10:08:05.908 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone3", 2021-09-17T10:08:05.908 INFO:teuthology.orchestra.run.smithi018.stdout: "domain_root": "test-zone3.rgw.meta:root", 2021-09-17T10:08:05.909 INFO:teuthology.orchestra.run.smithi018.stdout: "control_pool": "test-zone3.rgw.control", 2021-09-17T10:08:05.909 INFO:teuthology.orchestra.run.smithi018.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2021-09-17T10:08:05.909 INFO:teuthology.orchestra.run.smithi018.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2021-09-17T10:08:05.909 INFO:teuthology.orchestra.run.smithi018.stdout: "log_pool": "test-zone3.rgw.log", 2021-09-17T10:08:05.910 INFO:teuthology.orchestra.run.smithi018.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2021-09-17T10:08:05.910 INFO:teuthology.orchestra.run.smithi018.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2021-09-17T10:08:05.910 INFO:teuthology.orchestra.run.smithi018.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2021-09-17T10:08:05.910 INFO:teuthology.orchestra.run.smithi018.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2021-09-17T10:08:05.910 INFO:teuthology.orchestra.run.smithi018.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2021-09-17T10:08:05.911 INFO:teuthology.orchestra.run.smithi018.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2021-09-17T10:08:05.911 INFO:teuthology.orchestra.run.smithi018.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2021-09-17T10:08:05.911 INFO:teuthology.orchestra.run.smithi018.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2021-09-17T10:08:05.911 INFO:teuthology.orchestra.run.smithi018.stdout: "otp_pool": "test-zone3.rgw.otp", 2021-09-17T10:08:05.912 INFO:teuthology.orchestra.run.smithi018.stdout: "system_key": { 2021-09-17T10:08:05.912 INFO:teuthology.orchestra.run.smithi018.stdout: "access_key": "H92RM4Y9KBWIE4BC", 2021-09-17T10:08:05.912 INFO:teuthology.orchestra.run.smithi018.stdout: "secret_key": "RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK" 2021-09-17T10:08:05.912 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:05.912 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_pools": [ 2021-09-17T10:08:05.913 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:05.913 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "default-placement", 2021-09-17T10:08:05.913 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:08:05.913 INFO:teuthology.orchestra.run.smithi018.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2021-09-17T10:08:05.914 INFO:teuthology.orchestra.run.smithi018.stdout: "storage_classes": { 2021-09-17T10:08:05.914 INFO:teuthology.orchestra.run.smithi018.stdout: "STANDARD": { 2021-09-17T10:08:05.914 INFO:teuthology.orchestra.run.smithi018.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2021-09-17T10:08:05.914 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:05.914 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:05.915 INFO:teuthology.orchestra.run.smithi018.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2021-09-17T10:08:05.915 INFO:teuthology.orchestra.run.smithi018.stdout: "index_type": 0 2021-09-17T10:08:05.915 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:05.915 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:05.915 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:08:05.916 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:08:05.916 INFO:teuthology.orchestra.run.smithi018.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2021-09-17T10:08:05.916 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:08:05.917 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-09-17T10:08:07.179 INFO:teuthology.orchestra.run.smithi018.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-09-17T10:08:07.194 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-09-17T10:08:08.193 INFO:teuthology.orchestra.run.smithi018.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-09-17T10:08:08.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:08.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:08.211 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:08:08.260 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:08.260 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:08.315 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:08:08.316 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "d9facd38-bb4e-4ea2-9817-2655cc5c6379", 2021-09-17T10:08:08.316 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone3", 2021-09-17T10:08:08.316 INFO:teuthology.orchestra.run.smithi018.stdout: "domain_root": "test-zone3.rgw.meta:root", 2021-09-17T10:08:08.316 INFO:teuthology.orchestra.run.smithi018.stdout: "control_pool": "test-zone3.rgw.control", 2021-09-17T10:08:08.316 INFO:teuthology.orchestra.run.smithi018.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2021-09-17T10:08:08.317 INFO:teuthology.orchestra.run.smithi018.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2021-09-17T10:08:08.317 INFO:teuthology.orchestra.run.smithi018.stdout: "log_pool": "test-zone3.rgw.log", 2021-09-17T10:08:08.317 INFO:teuthology.orchestra.run.smithi018.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2021-09-17T10:08:08.317 INFO:teuthology.orchestra.run.smithi018.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2021-09-17T10:08:08.318 INFO:teuthology.orchestra.run.smithi018.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2021-09-17T10:08:08.318 INFO:teuthology.orchestra.run.smithi018.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2021-09-17T10:08:08.318 INFO:teuthology.orchestra.run.smithi018.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2021-09-17T10:08:08.318 INFO:teuthology.orchestra.run.smithi018.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2021-09-17T10:08:08.318 INFO:teuthology.orchestra.run.smithi018.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2021-09-17T10:08:08.319 INFO:teuthology.orchestra.run.smithi018.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2021-09-17T10:08:08.319 INFO:teuthology.orchestra.run.smithi018.stdout: "otp_pool": "test-zone3.rgw.otp", 2021-09-17T10:08:08.319 INFO:teuthology.orchestra.run.smithi018.stdout: "system_key": { 2021-09-17T10:08:08.319 INFO:teuthology.orchestra.run.smithi018.stdout: "access_key": "H92RM4Y9KBWIE4BC", 2021-09-17T10:08:08.320 INFO:teuthology.orchestra.run.smithi018.stdout: "secret_key": "RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK" 2021-09-17T10:08:08.320 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:08.320 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_pools": [ 2021-09-17T10:08:08.320 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:08.321 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "default-placement", 2021-09-17T10:08:08.321 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:08:08.321 INFO:teuthology.orchestra.run.smithi018.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2021-09-17T10:08:08.321 INFO:teuthology.orchestra.run.smithi018.stdout: "storage_classes": { 2021-09-17T10:08:08.321 INFO:teuthology.orchestra.run.smithi018.stdout: "STANDARD": { 2021-09-17T10:08:08.322 INFO:teuthology.orchestra.run.smithi018.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2021-09-17T10:08:08.322 INFO:teuthology.orchestra.run.smithi018.stdout: "compression_type": "random" 2021-09-17T10:08:08.322 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:08.322 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:08.323 INFO:teuthology.orchestra.run.smithi018.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2021-09-17T10:08:08.323 INFO:teuthology.orchestra.run.smithi018.stdout: "index_type": 0 2021-09-17T10:08:08.323 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:08.323 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:08.323 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:08:08.324 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "", 2021-09-17T10:08:08.324 INFO:teuthology.orchestra.run.smithi018.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2021-09-17T10:08:08.324 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:08:08.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:08.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:08.326 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:08:08.367 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:08.367 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:08.386 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:08:08.400+0000 7f5a1fb85b00 0 period (3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac does not have zone d9facd38-bb4e-4ea2-9817-2655cc5c6379 configured 2021-09-17T10:08:08.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-09-17T10:08:16.310 INFO:teuthology.orchestra.run.smithi018.stderr:Sending period to new master zone 9854c5a3-6cc0-45a7-b530-3af2b1c733d5 2021-09-17T10:08:16.356 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:08:16.357 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:08:16.357 INFO:teuthology.orchestra.run.smithi018.stdout: "epoch": 3, 2021-09-17T10:08:16.357 INFO:teuthology.orchestra.run.smithi018.stdout: "predecessor_uuid": "20a9da20-85d6-489f-bcfe-ebc80031614a", 2021-09-17T10:08:16.358 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": [], 2021-09-17T10:08:16.358 INFO:teuthology.orchestra.run.smithi018.stdout: "period_map": { 2021-09-17T10:08:16.358 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:08:16.358 INFO:teuthology.orchestra.run.smithi018.stdout: "zonegroups": [ 2021-09-17T10:08:16.359 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:16.359 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "test-zonegroup", 2021-09-17T10:08:16.359 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zonegroup", 2021-09-17T10:08:16.359 INFO:teuthology.orchestra.run.smithi018.stdout: "api_name": "", 2021-09-17T10:08:16.360 INFO:teuthology.orchestra.run.smithi018.stdout: "is_master": "true", 2021-09-17T10:08:16.360 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:08:16.360 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:08:16.360 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:08:16.360 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames": [], 2021-09-17T10:08:16.361 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames_s3website": [], 2021-09-17T10:08:16.361 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:08:16.361 INFO:teuthology.orchestra.run.smithi018.stdout: "zones": [ 2021-09-17T10:08:16.362 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:16.362 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:08:16.362 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone2", 2021-09-17T10:08:16.363 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:08:16.363 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi050.front.sepia.ceph.com:8000" 2021-09-17T10:08:16.363 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:08:16.364 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:08:16.364 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "true", 2021-09-17T10:08:16.364 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:08:16.364 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:08:16.365 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "", 2021-09-17T10:08:16.365 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "true", 2021-09-17T10:08:16.365 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [], 2021-09-17T10:08:16.365 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:08:16.366 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:16.366 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:16.366 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:08:16.366 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone1", 2021-09-17T10:08:16.367 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:08:16.367 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:08:16.367 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:08:16.367 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:08:16.368 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "true", 2021-09-17T10:08:16.368 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:08:16.368 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:08:16.369 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "", 2021-09-17T10:08:16.369 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "true", 2021-09-17T10:08:16.369 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [], 2021-09-17T10:08:16.369 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:08:16.370 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:16.370 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:16.370 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "d9facd38-bb4e-4ea2-9817-2655cc5c6379", 2021-09-17T10:08:16.371 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone3", 2021-09-17T10:08:16.371 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:08:16.371 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi018.front.sepia.ceph.com:8001" 2021-09-17T10:08:16.372 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:08:16.372 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:08:16.372 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "true", 2021-09-17T10:08:16.373 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:08:16.373 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:08:16.373 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "", 2021-09-17T10:08:16.374 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "true", 2021-09-17T10:08:16.374 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [], 2021-09-17T10:08:16.374 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:08:16.375 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:16.375 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:08:16.375 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_targets": [ 2021-09-17T10:08:16.375 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:16.376 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "default-placement", 2021-09-17T10:08:16.376 INFO:teuthology.orchestra.run.smithi018.stdout: "tags": [], 2021-09-17T10:08:16.376 INFO:teuthology.orchestra.run.smithi018.stdout: "storage_classes": [ 2021-09-17T10:08:16.376 INFO:teuthology.orchestra.run.smithi018.stdout: "STANDARD" 2021-09-17T10:08:16.377 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:08:16.377 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:16.377 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:08:16.377 INFO:teuthology.orchestra.run.smithi018.stdout: "default_placement": "default-placement", 2021-09-17T10:08:16.378 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:08:16.378 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_policy": { 2021-09-17T10:08:16.378 INFO:teuthology.orchestra.run.smithi018.stdout: "groups": [] 2021-09-17T10:08:16.378 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:16.379 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:16.379 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:08:16.379 INFO:teuthology.orchestra.run.smithi018.stdout: "short_zone_ids": [ 2021-09-17T10:08:16.379 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:16.379 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:08:16.380 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 1815852058 2021-09-17T10:08:16.380 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:16.380 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:16.381 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:08:16.381 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 664423460 2021-09-17T10:08:16.381 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:16.381 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:16.382 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "d9facd38-bb4e-4ea2-9817-2655cc5c6379", 2021-09-17T10:08:16.382 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 2096482420 2021-09-17T10:08:16.382 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:16.382 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:08:16.383 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:16.383 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T10:08:16.383 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:08:16.383 INFO:teuthology.orchestra.run.smithi018.stdout: "period_config": { 2021-09-17T10:08:16.384 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_quota": { 2021-09-17T10:08:16.384 INFO:teuthology.orchestra.run.smithi018.stdout: "enabled": false, 2021-09-17T10:08:16.384 INFO:teuthology.orchestra.run.smithi018.stdout: "check_on_raw": false, 2021-09-17T10:08:16.385 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size": -1, 2021-09-17T10:08:16.385 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size_kb": 0, 2021-09-17T10:08:16.385 INFO:teuthology.orchestra.run.smithi018.stdout: "max_objects": -1 2021-09-17T10:08:16.385 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:16.385 INFO:teuthology.orchestra.run.smithi018.stdout: "user_quota": { 2021-09-17T10:08:16.386 INFO:teuthology.orchestra.run.smithi018.stdout: "enabled": false, 2021-09-17T10:08:16.386 INFO:teuthology.orchestra.run.smithi018.stdout: "check_on_raw": false, 2021-09-17T10:08:16.388 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size": -1, 2021-09-17T10:08:16.388 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size_kb": 0, 2021-09-17T10:08:16.388 INFO:teuthology.orchestra.run.smithi018.stdout: "max_objects": -1 2021-09-17T10:08:16.389 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:16.389 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:16.389 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:08:16.389 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_name": "test-realm", 2021-09-17T10:08:16.390 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:08:16.390 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:08:16.391 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-09-17T10:08:16.391 INFO:teuthology.orchestra.run:waiting for 300 2021-09-17T10:08:16.392 INFO:tasks.rgw.c1.client.1.smithi018.stdout:2021-09-17T10:08:16.408+0000 7f150e499700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 17987) UID: 0 2021-09-17T10:08:16.392 INFO:tasks.rgw.c1.client.1.smithi018.stdout:2021-09-17T10:08:16.408+0000 7f1512964b40 -1 shutting down 2021-09-17T10:08:16.858 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-09-17T10:08:22.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~6s 2021-09-17T10:08:22.494 INFO:tasks.rgw.c1.client.1:Stopped 2021-09-17T10:08:22.494 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-09-17T10:08:22.495 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-09-17T10:08:22.497 INFO:tasks.rgw.c1.client.1:Started 2021-09-17T10:08:22.498 DEBUG:teuthology.orchestra.run.smithi018:> curl http://smithi018.front.sepia.ceph.com:8001 2021-09-17T10:08:22.513 INFO:teuthology.orchestra.run.smithi018.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:08:22.513 INFO:teuthology.orchestra.run.smithi018.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:08:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-17T10:08:22.515 INFO:teuthology.orchestra.run.smithi018.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2021-09-17T10:08:22.515 INFO:teuthology.orchestra.run.smithi018.stderr:curl: (7) Failed to connect to smithi018.front.sepia.ceph.com port 8001: Connection refused 2021-09-17T10:08:23.517 DEBUG:teuthology.orchestra.run.smithi018:> curl http://smithi018.front.sepia.ceph.com:8001 2021-09-17T10:08:23.532 INFO:teuthology.orchestra.run.smithi018.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:08:23.533 INFO:teuthology.orchestra.run.smithi018.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:08:23.535 INFO:teuthology.orchestra.run.smithi018.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2021-09-17T10:08:23.536 INFO:teuthology.orchestra.run.smithi018.stdout:anonymous 2021-09-17T10:08:23.537 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-09-17T10:08:23.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi018.front.sepia.ceph.com:8000', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--url', 'http://smithi018.front.sepia.ceph.com:8000', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:23.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi018.front.sepia.ceph.com:8000', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--url', 'http://smithi018.front.sepia.ceph.com:8000', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:23.538 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi018.front.sepia.ceph.com:8000 --access-key H92RM4Y9KBWIE4BC --secret RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK --url http://smithi018.front.sepia.ceph.com:8000 --access-key H92RM4Y9KBWIE4BC --secret RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T10:08:23.588 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:23.589 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:23.660 INFO:tasks.rgw.c2.client.0.smithi050.stdout:2021-09-17T10:08:23.674+0000 7fd333e07700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.3c1ea6e8-af17-477a-9c20-483e62be612c.control err=-107 2021-09-17T10:08:23.680 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:08:23.681 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:08:23.681 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-realm", 2021-09-17T10:08:23.681 INFO:teuthology.orchestra.run.smithi050.stdout: "current_period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:08:23.682 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 2 2021-09-17T10:08:23.682 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:08:23.684 INFO:tasks.rgw_multisite:creating zone.. 2021-09-17T10:08:23.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--endpoints', 'http://smithi050.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:23.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'H92RM4Y9KBWIE4BC', '--secret', 'RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK', '--endpoints', 'http://smithi050.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:23.685 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key H92RM4Y9KBWIE4BC --secret RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK --endpoints http://smithi050.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T10:08:23.737 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:23.737 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:08:23.814+0000 7fb5e34e1b00 0 failed reading obj info from .rgw.root:zone_info.9854c5a3-6cc0-45a7-b530-3af2b1c733d5: (2) No such file or directory 2021-09-17T10:08:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:08:23.814+0000 7fb5e34e1b00 0 WARNING: could not read zone params for zone id=9854c5a3-6cc0-45a7-b530-3af2b1c733d5 name=test-zone1 2021-09-17T10:08:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:08:23.814+0000 7fb5e34e1b00 0 failed reading obj info from .rgw.root:zone_info.d9facd38-bb4e-4ea2-9817-2655cc5c6379: (2) No such file or directory 2021-09-17T10:08:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:08:23.814+0000 7fb5e34e1b00 0 WARNING: could not read zone params for zone id=d9facd38-bb4e-4ea2-9817-2655cc5c6379 name=test-zone3 2021-09-17T10:08:23.810 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:08:23.810 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "74241c6a-1297-4f4a-8bbd-b65c661636d9", 2021-09-17T10:08:23.810 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone4", 2021-09-17T10:08:23.810 INFO:teuthology.orchestra.run.smithi050.stdout: "domain_root": "test-zone4.rgw.meta:root", 2021-09-17T10:08:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: "control_pool": "test-zone4.rgw.control", 2021-09-17T10:08:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2021-09-17T10:08:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2021-09-17T10:08:23.811 INFO:teuthology.orchestra.run.smithi050.stdout: "log_pool": "test-zone4.rgw.log", 2021-09-17T10:08:23.812 INFO:teuthology.orchestra.run.smithi050.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2021-09-17T10:08:23.812 INFO:teuthology.orchestra.run.smithi050.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2021-09-17T10:08:23.812 INFO:teuthology.orchestra.run.smithi050.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2021-09-17T10:08:23.812 INFO:teuthology.orchestra.run.smithi050.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2021-09-17T10:08:23.812 INFO:teuthology.orchestra.run.smithi050.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2021-09-17T10:08:23.813 INFO:teuthology.orchestra.run.smithi050.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2021-09-17T10:08:23.813 INFO:teuthology.orchestra.run.smithi050.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2021-09-17T10:08:23.813 INFO:teuthology.orchestra.run.smithi050.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2021-09-17T10:08:23.813 INFO:teuthology.orchestra.run.smithi050.stdout: "otp_pool": "test-zone4.rgw.otp", 2021-09-17T10:08:23.814 INFO:teuthology.orchestra.run.smithi050.stdout: "system_key": { 2021-09-17T10:08:23.814 INFO:teuthology.orchestra.run.smithi050.stdout: "access_key": "H92RM4Y9KBWIE4BC", 2021-09-17T10:08:23.814 INFO:teuthology.orchestra.run.smithi050.stdout: "secret_key": "RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK" 2021-09-17T10:08:23.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:23.814 INFO:teuthology.orchestra.run.smithi050.stdout: "placement_pools": [ 2021-09-17T10:08:23.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:23.815 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "default-placement", 2021-09-17T10:08:23.815 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:23.815 INFO:teuthology.orchestra.run.smithi050.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2021-09-17T10:08:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_classes": { 2021-09-17T10:08:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: "STANDARD": { 2021-09-17T10:08:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2021-09-17T10:08:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:23.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:23.817 INFO:teuthology.orchestra.run.smithi050.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2021-09-17T10:08:23.817 INFO:teuthology.orchestra.run.smithi050.stdout: "index_type": 0 2021-09-17T10:08:23.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:23.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:23.822 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:08:23.823 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_config": { 2021-09-17T10:08:23.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event_retention_days": 7, 2021-09-17T10:08:23.823 INFO:teuthology.orchestra.run.smithi050.stdout: "start_with_full_sync": false 2021-09-17T10:08:23.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:23.823 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:08:23.824 INFO:teuthology.orchestra.run.smithi050.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2021-09-17T10:08:23.824 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:08:23.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-09-17T10:08:24.309 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-09-17T10:08:24.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-09-17T10:08:25.311 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-09-17T10:08:25.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:25.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:25.337 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T10:08:25.373 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:25.374 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:25.425 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:08:25.425 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "74241c6a-1297-4f4a-8bbd-b65c661636d9", 2021-09-17T10:08:25.426 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone4", 2021-09-17T10:08:25.426 INFO:teuthology.orchestra.run.smithi050.stdout: "domain_root": "test-zone4.rgw.meta:root", 2021-09-17T10:08:25.426 INFO:teuthology.orchestra.run.smithi050.stdout: "control_pool": "test-zone4.rgw.control", 2021-09-17T10:08:25.426 INFO:teuthology.orchestra.run.smithi050.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2021-09-17T10:08:25.427 INFO:teuthology.orchestra.run.smithi050.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2021-09-17T10:08:25.427 INFO:teuthology.orchestra.run.smithi050.stdout: "log_pool": "test-zone4.rgw.log", 2021-09-17T10:08:25.427 INFO:teuthology.orchestra.run.smithi050.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2021-09-17T10:08:25.427 INFO:teuthology.orchestra.run.smithi050.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2021-09-17T10:08:25.428 INFO:teuthology.orchestra.run.smithi050.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2021-09-17T10:08:25.428 INFO:teuthology.orchestra.run.smithi050.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2021-09-17T10:08:25.428 INFO:teuthology.orchestra.run.smithi050.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2021-09-17T10:08:25.428 INFO:teuthology.orchestra.run.smithi050.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2021-09-17T10:08:25.428 INFO:teuthology.orchestra.run.smithi050.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2021-09-17T10:08:25.429 INFO:teuthology.orchestra.run.smithi050.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2021-09-17T10:08:25.429 INFO:teuthology.orchestra.run.smithi050.stdout: "otp_pool": "test-zone4.rgw.otp", 2021-09-17T10:08:25.429 INFO:teuthology.orchestra.run.smithi050.stdout: "system_key": { 2021-09-17T10:08:25.429 INFO:teuthology.orchestra.run.smithi050.stdout: "access_key": "H92RM4Y9KBWIE4BC", 2021-09-17T10:08:25.430 INFO:teuthology.orchestra.run.smithi050.stdout: "secret_key": "RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK" 2021-09-17T10:08:25.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:25.430 INFO:teuthology.orchestra.run.smithi050.stdout: "placement_pools": [ 2021-09-17T10:08:25.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:25.430 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "default-placement", 2021-09-17T10:08:25.431 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:25.431 INFO:teuthology.orchestra.run.smithi050.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2021-09-17T10:08:25.431 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_classes": { 2021-09-17T10:08:25.431 INFO:teuthology.orchestra.run.smithi050.stdout: "STANDARD": { 2021-09-17T10:08:25.432 INFO:teuthology.orchestra.run.smithi050.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2021-09-17T10:08:25.432 INFO:teuthology.orchestra.run.smithi050.stdout: "compression_type": "random" 2021-09-17T10:08:25.432 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:25.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:25.432 INFO:teuthology.orchestra.run.smithi050.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2021-09-17T10:08:25.433 INFO:teuthology.orchestra.run.smithi050.stdout: "index_type": 0 2021-09-17T10:08:25.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:25.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:25.433 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:08:25.434 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_config": { 2021-09-17T10:08:25.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event_retention_days": 7, 2021-09-17T10:08:25.434 INFO:teuthology.orchestra.run.smithi050.stdout: "start_with_full_sync": false 2021-09-17T10:08:25.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:25.434 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "", 2021-09-17T10:08:25.435 INFO:teuthology.orchestra.run.smithi050.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2021-09-17T10:08:25.435 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:08:25.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:25.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:25.437 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T10:08:25.519 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:25.519 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:25.539 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:08:25.554+0000 7fe32a185b00 0 period (3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac does not have zone 74241c6a-1297-4f4a-8bbd-b65c661636d9 configured 2021-09-17T10:08:28.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-09-17T10:08:33.400 INFO:teuthology.orchestra.run.smithi050.stderr:Sending period to new master zone 9854c5a3-6cc0-45a7-b530-3af2b1c733d5 2021-09-17T10:08:33.453 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:08:33.454 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:08:33.454 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 4, 2021-09-17T10:08:33.454 INFO:teuthology.orchestra.run.smithi050.stdout: "predecessor_uuid": "20a9da20-85d6-489f-bcfe-ebc80031614a", 2021-09-17T10:08:33.455 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": [], 2021-09-17T10:08:33.455 INFO:teuthology.orchestra.run.smithi050.stdout: "period_map": { 2021-09-17T10:08:33.456 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:08:33.456 INFO:teuthology.orchestra.run.smithi050.stdout: "zonegroups": [ 2021-09-17T10:08:33.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:33.456 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "test-zonegroup", 2021-09-17T10:08:33.457 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zonegroup", 2021-09-17T10:08:33.457 INFO:teuthology.orchestra.run.smithi050.stdout: "api_name": "", 2021-09-17T10:08:33.457 INFO:teuthology.orchestra.run.smithi050.stdout: "is_master": "true", 2021-09-17T10:08:33.457 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:08:33.458 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:08:33.458 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:08:33.458 INFO:teuthology.orchestra.run.smithi050.stdout: "hostnames": [], 2021-09-17T10:08:33.458 INFO:teuthology.orchestra.run.smithi050.stdout: "hostnames_s3website": [], 2021-09-17T10:08:33.458 INFO:teuthology.orchestra.run.smithi050.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:08:33.459 INFO:teuthology.orchestra.run.smithi050.stdout: "zones": [ 2021-09-17T10:08:33.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:33.459 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:08:33.459 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone2", 2021-09-17T10:08:33.460 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:08:33.460 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi050.front.sepia.ceph.com:8000" 2021-09-17T10:08:33.460 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:08:33.460 INFO:teuthology.orchestra.run.smithi050.stdout: "log_meta": "false", 2021-09-17T10:08:33.461 INFO:teuthology.orchestra.run.smithi050.stdout: "log_data": "true", 2021-09-17T10:08:33.461 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:08:33.461 INFO:teuthology.orchestra.run.smithi050.stdout: "read_only": "false", 2021-09-17T10:08:33.462 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_type": "", 2021-09-17T10:08:33.462 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from_all": "true", 2021-09-17T10:08:33.462 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from": [], 2021-09-17T10:08:33.462 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_zone": "" 2021-09-17T10:08:33.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:33.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:33.463 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "74241c6a-1297-4f4a-8bbd-b65c661636d9", 2021-09-17T10:08:33.463 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone4", 2021-09-17T10:08:33.463 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:08:33.464 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi050.front.sepia.ceph.com:8001" 2021-09-17T10:08:33.464 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:08:33.464 INFO:teuthology.orchestra.run.smithi050.stdout: "log_meta": "false", 2021-09-17T10:08:33.464 INFO:teuthology.orchestra.run.smithi050.stdout: "log_data": "true", 2021-09-17T10:08:33.465 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:08:33.465 INFO:teuthology.orchestra.run.smithi050.stdout: "read_only": "false", 2021-09-17T10:08:33.465 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_type": "pubsub", 2021-09-17T10:08:33.465 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from_all": "false", 2021-09-17T10:08:33.466 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from": [ 2021-09-17T10:08:33.466 INFO:teuthology.orchestra.run.smithi050.stdout: "test-zone1" 2021-09-17T10:08:33.466 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:08:33.466 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_zone": "" 2021-09-17T10:08:33.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:33.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:33.467 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:08:33.467 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone1", 2021-09-17T10:08:33.467 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:08:33.468 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:08:33.468 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:08:33.468 INFO:teuthology.orchestra.run.smithi050.stdout: "log_meta": "false", 2021-09-17T10:08:33.468 INFO:teuthology.orchestra.run.smithi050.stdout: "log_data": "true", 2021-09-17T10:08:33.469 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:08:33.469 INFO:teuthology.orchestra.run.smithi050.stdout: "read_only": "false", 2021-09-17T10:08:33.469 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_type": "", 2021-09-17T10:08:33.469 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from_all": "true", 2021-09-17T10:08:33.470 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from": [], 2021-09-17T10:08:33.470 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_zone": "" 2021-09-17T10:08:33.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:33.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:33.471 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "d9facd38-bb4e-4ea2-9817-2655cc5c6379", 2021-09-17T10:08:33.471 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone3", 2021-09-17T10:08:33.471 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:08:33.472 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi018.front.sepia.ceph.com:8001" 2021-09-17T10:08:33.472 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:08:33.472 INFO:teuthology.orchestra.run.smithi050.stdout: "log_meta": "false", 2021-09-17T10:08:33.472 INFO:teuthology.orchestra.run.smithi050.stdout: "log_data": "true", 2021-09-17T10:08:33.473 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:08:33.473 INFO:teuthology.orchestra.run.smithi050.stdout: "read_only": "false", 2021-09-17T10:08:33.473 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_type": "", 2021-09-17T10:08:33.473 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from_all": "true", 2021-09-17T10:08:33.474 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from": [], 2021-09-17T10:08:33.474 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_zone": "" 2021-09-17T10:08:33.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:33.474 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:08:33.475 INFO:teuthology.orchestra.run.smithi050.stdout: "placement_targets": [ 2021-09-17T10:08:33.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:33.475 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "default-placement", 2021-09-17T10:08:33.475 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2021-09-17T10:08:33.476 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_classes": [ 2021-09-17T10:08:33.476 INFO:teuthology.orchestra.run.smithi050.stdout: "STANDARD" 2021-09-17T10:08:33.476 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:08:33.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:33.477 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:08:33.477 INFO:teuthology.orchestra.run.smithi050.stdout: "default_placement": "default-placement", 2021-09-17T10:08:33.477 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:08:33.477 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_policy": { 2021-09-17T10:08:33.478 INFO:teuthology.orchestra.run.smithi050.stdout: "groups": [] 2021-09-17T10:08:33.478 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:33.478 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:33.478 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:08:33.479 INFO:teuthology.orchestra.run.smithi050.stdout: "short_zone_ids": [ 2021-09-17T10:08:33.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:33.480 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:08:33.481 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 1815852058 2021-09-17T10:08:33.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:33.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:33.481 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "74241c6a-1297-4f4a-8bbd-b65c661636d9", 2021-09-17T10:08:33.481 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 1495243106 2021-09-17T10:08:33.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:33.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:33.482 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:08:33.482 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 664423460 2021-09-17T10:08:33.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:33.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:33.483 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "d9facd38-bb4e-4ea2-9817-2655cc5c6379", 2021-09-17T10:08:33.483 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 2096482420 2021-09-17T10:08:33.484 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:33.484 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:08:33.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:33.484 INFO:teuthology.orchestra.run.smithi050.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T10:08:33.485 INFO:teuthology.orchestra.run.smithi050.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:08:33.485 INFO:teuthology.orchestra.run.smithi050.stdout: "period_config": { 2021-09-17T10:08:33.485 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_quota": { 2021-09-17T10:08:33.485 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": false, 2021-09-17T10:08:33.486 INFO:teuthology.orchestra.run.smithi050.stdout: "check_on_raw": false, 2021-09-17T10:08:33.486 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size": -1, 2021-09-17T10:08:33.486 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size_kb": 0, 2021-09-17T10:08:33.486 INFO:teuthology.orchestra.run.smithi050.stdout: "max_objects": -1 2021-09-17T10:08:33.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:33.487 INFO:teuthology.orchestra.run.smithi050.stdout: "user_quota": { 2021-09-17T10:08:33.487 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": false, 2021-09-17T10:08:33.487 INFO:teuthology.orchestra.run.smithi050.stdout: "check_on_raw": false, 2021-09-17T10:08:33.487 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size": -1, 2021-09-17T10:08:33.488 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size_kb": 0, 2021-09-17T10:08:33.488 INFO:teuthology.orchestra.run.smithi050.stdout: "max_objects": -1 2021-09-17T10:08:33.488 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:33.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:33.489 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:08:33.489 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_name": "test-realm", 2021-09-17T10:08:33.489 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:33.489 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:08:33.490 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-09-17T10:08:33.490 INFO:teuthology.orchestra.run:waiting for 300 2021-09-17T10:08:33.491 INFO:tasks.rgw.c2.client.1.smithi050.stdout:2021-09-17T10:08:33.506+0000 7fe86a72e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 18340) UID: 0 2021-09-17T10:08:33.491 INFO:tasks.rgw.c2.client.1.smithi050.stdout:2021-09-17T10:08:33.506+0000 7fe86ebf9b40 -1 shutting down 2021-09-17T10:08:37.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-09-17T10:08:39.593 INFO:tasks.rgw.c2.client.1:Stopped 2021-09-17T10:08:39.593 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-09-17T10:08:39.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-09-17T10:08:39.597 INFO:tasks.rgw.c2.client.1:Started 2021-09-17T10:08:39.597 DEBUG:teuthology.orchestra.run.smithi050:> curl http://smithi050.front.sepia.ceph.com:8001 2021-09-17T10:08:39.615 INFO:teuthology.orchestra.run.smithi050.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:08:39.616 INFO:teuthology.orchestra.run.smithi050.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:08:39.616 INFO:teuthology.orchestra.run.smithi050.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2021-09-17T10:08:39.617 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-17T10:08:39.618 INFO:teuthology.orchestra.run.smithi050.stderr:curl: (7) Failed to connect to smithi050.front.sepia.ceph.com port 8001: Connection refused 2021-09-17T10:08:40.619 DEBUG:teuthology.orchestra.run.smithi050:> curl http://smithi050.front.sepia.ceph.com:8001 2021-09-17T10:08:40.635 INFO:teuthology.orchestra.run.smithi050.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-17T10:08:40.636 INFO:teuthology.orchestra.run.smithi050.stderr: Dload Upload Total Spent Left Speed 2021-09-17T10:08:40.637 INFO:teuthology.orchestra.run.smithi050.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 130k 0 --:--:-- --:--:-- --:--:-- 130k 2021-09-17T10:08:40.638 INFO:teuthology.orchestra.run.smithi050.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-0061446928-10ea-test-zone4","HostId":"10ea-test-zone4-test-zonegroup"} 2021-09-17T10:08:40.638 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-09-17T10:08:40.639 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-09-17T10:08:40.655 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/git.ceph.com_ceph-c_f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/qa/../src/test/rgw/rgw_multi 2021-09-17T10:08:40.670 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-09-17T10:08:40.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:40.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:08:40.672 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:08:40.718 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:40.718 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:40.814 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:08:40.814 INFO:teuthology.orchestra.run.smithi018.stdout: "user_id": "rgw-multisite-test-user", 2021-09-17T10:08:40.815 INFO:teuthology.orchestra.run.smithi018.stdout: "display_name": "Multisite Test User", 2021-09-17T10:08:40.815 INFO:teuthology.orchestra.run.smithi018.stdout: "email": "", 2021-09-17T10:08:40.815 INFO:teuthology.orchestra.run.smithi018.stdout: "suspended": 0, 2021-09-17T10:08:40.815 INFO:teuthology.orchestra.run.smithi018.stdout: "max_buckets": 1000, 2021-09-17T10:08:40.816 INFO:teuthology.orchestra.run.smithi018.stdout: "subusers": [], 2021-09-17T10:08:40.816 INFO:teuthology.orchestra.run.smithi018.stdout: "keys": [ 2021-09-17T10:08:40.816 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:40.816 INFO:teuthology.orchestra.run.smithi018.stdout: "user": "rgw-multisite-test-user", 2021-09-17T10:08:40.816 INFO:teuthology.orchestra.run.smithi018.stdout: "access_key": "OB5G3AXRMFUTD02L23O1", 2021-09-17T10:08:40.817 INFO:teuthology.orchestra.run.smithi018.stdout: "secret_key": "ziXmrQFupUCw87yW2SfJzfPYYZHwnypQBKWwB5eb" 2021-09-17T10:08:40.817 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:40.817 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:08:40.817 INFO:teuthology.orchestra.run.smithi018.stdout: "swift_keys": [], 2021-09-17T10:08:40.818 INFO:teuthology.orchestra.run.smithi018.stdout: "caps": [], 2021-09-17T10:08:40.818 INFO:teuthology.orchestra.run.smithi018.stdout: "op_mask": "read, write, delete", 2021-09-17T10:08:40.818 INFO:teuthology.orchestra.run.smithi018.stdout: "default_placement": "", 2021-09-17T10:08:40.818 INFO:teuthology.orchestra.run.smithi018.stdout: "default_storage_class": "", 2021-09-17T10:08:40.819 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_tags": [], 2021-09-17T10:08:40.819 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_quota": { 2021-09-17T10:08:40.819 INFO:teuthology.orchestra.run.smithi018.stdout: "enabled": false, 2021-09-17T10:08:40.819 INFO:teuthology.orchestra.run.smithi018.stdout: "check_on_raw": false, 2021-09-17T10:08:40.819 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size": -1, 2021-09-17T10:08:40.820 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size_kb": 0, 2021-09-17T10:08:40.820 INFO:teuthology.orchestra.run.smithi018.stdout: "max_objects": -1 2021-09-17T10:08:40.820 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:40.820 INFO:teuthology.orchestra.run.smithi018.stdout: "user_quota": { 2021-09-17T10:08:40.820 INFO:teuthology.orchestra.run.smithi018.stdout: "enabled": false, 2021-09-17T10:08:40.821 INFO:teuthology.orchestra.run.smithi018.stdout: "check_on_raw": false, 2021-09-17T10:08:40.821 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size": -1, 2021-09-17T10:08:40.821 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size_kb": 0, 2021-09-17T10:08:40.821 INFO:teuthology.orchestra.run.smithi018.stdout: "max_objects": -1 2021-09-17T10:08:40.822 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:40.822 INFO:teuthology.orchestra.run.smithi018.stdout: "temp_url_keys": [], 2021-09-17T10:08:40.822 INFO:teuthology.orchestra.run.smithi018.stdout: "type": "rgw", 2021-09-17T10:08:40.822 INFO:teuthology.orchestra.run.smithi018.stdout: "mfa_ids": [] 2021-09-17T10:08:40.822 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:08:40.823 INFO:teuthology.orchestra.run.smithi018.stdout: 2021-09-17T10:08:40.842 INFO:rgw_multi.tests:meta checkpoint 2021-09-17T10:08:40.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:08:40.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:08:40.843 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:08:40.884 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:40.885 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:40.973 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:08:41.002 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:08:41.003 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:41.003 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873320.827024_49.1", 2021-09-17T10:08:41.003 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:08:40.827024Z" 2021-09-17T10:08:41.004 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:41.004 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:41.004 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:08:41.004 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:08:41.004 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:41.005 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:41.005 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:08:41.005 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:08:41.005 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:08:41.006 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:08:41.006 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:08:41.006 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:08:41.006 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:08:41.007 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:08:41.008 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:08:41.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:08:41.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:08:41.009 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:08:41.052 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:41.053 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:41.169 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:08:41.169 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:08:41.169 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:08:41.169 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:08:41.170 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:08:41.170 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:08:41.170 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:41.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:41.171 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:08:41.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:41.171 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:08:41.171 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:41.171 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:41.172 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:41.172 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:41.172 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:08:41.172 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:41.173 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:41.173 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:41.173 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:41.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:41.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:41.174 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:08:41.174 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:41.174 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:41.174 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:41.175 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:41.175 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:08:41.175 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:41.175 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:41.175 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:41.176 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:41.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:41.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:41.176 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:08:41.176 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:41.177 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:41.177 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:41.177 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:41.177 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:08:41.178 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:41.178 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:41.178 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:41.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:41.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:41.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:41.179 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:08:41.179 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:41.179 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:41.179 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:41.180 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:41.180 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:08:41.180 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:41.180 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:41.181 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:41.181 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:41.181 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:41.181 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:08:41.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:41.182 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:08:41.182 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:08:41.182 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:08:41.182 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:41.182 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:08:41.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:08:42.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-09-17T10:08:46.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:08:46.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:08:46.187 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:08:46.229 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:46.229 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:46.345 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:08:46.346 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:08:46.346 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:08:46.346 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:08:46.347 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:08:46.347 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:08:46.347 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:46.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:46.348 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:08:46.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:46.348 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:08:46.348 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:46.348 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:46.349 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:46.349 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:46.349 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:08:46.349 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:46.350 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:46.350 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:46.350 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:46.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:46.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:46.351 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:08:46.351 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:46.351 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:46.351 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:46.352 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:46.352 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:08:46.352 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:46.352 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:46.353 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:46.353 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:46.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:46.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:46.354 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:08:46.354 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:46.354 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:46.354 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:46.355 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:46.355 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:08:46.355 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:46.355 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:46.356 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:46.356 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:46.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:46.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:46.357 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:08:46.357 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:46.357 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:46.357 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:46.357 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:46.358 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:08:46.358 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:46.358 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:46.358 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:46.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:46.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:46.359 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:08:46.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:46.359 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:08:46.360 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:08:46.360 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:08:46.360 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:46.360 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:08:46.362 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:08:51.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:08:51.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:08:51.364 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:08:51.405 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:51.406 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:51.521 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:08:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:08:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:08:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:08:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:08:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:08:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:08:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:08:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:08:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:08:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:08:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:08:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:08:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:08:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:08:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:08:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:08:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:08:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:08:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:51.537 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:08:51.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:08:56.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:08:56.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:08:56.540 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:08:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:08:56.586 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:08:56.704 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:08:56.704 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:08:56.704 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:08:56.705 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:08:56.705 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:08:56.705 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:08:56.705 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:56.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:56.706 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:08:56.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:56.706 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:08:56.707 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:56.707 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:56.707 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:56.707 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:56.708 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:08:56.708 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:56.708 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:56.708 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:56.708 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:56.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:56.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:56.709 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:08:56.709 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:56.710 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:56.710 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:56.710 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:56.710 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:08:56.711 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:56.711 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:56.711 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:56.711 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:56.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:56.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:56.712 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:08:56.712 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:56.712 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:56.712 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:56.713 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:56.713 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:08:56.713 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:56.713 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:56.714 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:56.714 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:56.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:56.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:08:56.715 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:08:56.715 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:08:56.715 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:08:56.715 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:08:56.715 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:08:56.716 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:08:56.716 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:08:56.716 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:08:56.716 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:08:56.716 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:56.717 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:56.717 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:08:56.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:08:56.717 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:08:56.717 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:08:56.718 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:08:56.718 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:08:56.718 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:08:56.719 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:09:01.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:01.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:01.769 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:09:01.842 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:09:01.842 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:09:01.958 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:09:01.959 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:09:01.959 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:09:01.959 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:09:01.960 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:09:01.960 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:09:01.960 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:01.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:01.961 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:09:01.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:01.961 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:09:01.961 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:01.962 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:01.962 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:01.962 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:01.962 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:09:01.962 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:01.963 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:01.963 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:01.965 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:01.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:01.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:01.967 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:09:01.967 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:01.968 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:01.969 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:01.969 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:01.970 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:01.970 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:01.971 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:01.972 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:01.972 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:01.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:01.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:01.974 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:09:01.974 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:01.975 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:01.977 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:01.978 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:01.978 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:01.979 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:01.979 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:01.980 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:01.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:01.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:01.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:01.983 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:09:01.984 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:01.984 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:01.985 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:01.985 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:01.985 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:01.986 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:01.986 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:01.987 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:01.987 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:01.987 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:01.987 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:09:01.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:01.988 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:09:01.988 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:09:01.988 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:09:01.989 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:01.989 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:09:01.991 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:09:06.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:06.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:06.994 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:09:07.036 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:09:07.036 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:09:07.151 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:09:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:09:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:09:07.152 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:09:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:09:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:09:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:07.153 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:09:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:09:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:07.154 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:09:07.155 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:07.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:09:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:07.157 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:07.158 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:07.159 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:09:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:07.160 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:07.161 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:07.162 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:09:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:07.163 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:07.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:09:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:09:07.165 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:09:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:09:07.166 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:07.166 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:09:07.167 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:09:12.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:12.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:12.170 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:09:12.217 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:09:12.217 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:09:12.332 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:09:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:09:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:09:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:09:12.333 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:09:12.334 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:09:12.334 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:12.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:12.334 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:09:12.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:12.335 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:09:12.335 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:12.335 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:12.335 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:09:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:12.336 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:12.337 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:09:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:12.338 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:12.339 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:09:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:12.340 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:12.341 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:09:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:12.343 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:12.344 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:12.345 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:12.345 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:12.345 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:09:12.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:12.346 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:09:12.346 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:09:12.346 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:09:12.346 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:12.347 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:09:12.348 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:09:17.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:17.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:17.350 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:09:17.393 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:09:17.393 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:09:17.505 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:09:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:09:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:09:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:09:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:09:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:09:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:09:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:09:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:09:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:09:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:09:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:09:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:09:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:09:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:09:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:09:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:17.521 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:09:17.522 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:09:22.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:22.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:22.526 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:09:22.569 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:09:22.570 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:09:22.684 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:09:22.685 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:09:22.685 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:09:22.685 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:09:22.685 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:09:22.686 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:09:22.686 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:22.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:22.687 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:09:22.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:22.689 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:09:22.691 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:22.691 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:22.691 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:22.691 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:22.692 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:09:22.692 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:22.692 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:22.692 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:22.692 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:22.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:22.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:22.693 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:09:22.693 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:22.694 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:22.694 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:22.694 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:22.694 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:22.695 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:22.695 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:22.695 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:22.695 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:22.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:22.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:22.696 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:09:22.696 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:22.696 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:22.697 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:22.697 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:22.697 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:22.697 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:22.698 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:22.698 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:22.698 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:22.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:22.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:22.699 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:09:22.699 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:22.699 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:22.699 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:22.700 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:22.700 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:22.700 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:22.700 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:22.700 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:22.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:22.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:22.701 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:09:22.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:22.702 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:09:22.702 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:09:22.702 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:09:22.702 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:22.703 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:09:22.704 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:09:27.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:27.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:27.706 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:09:27.749 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:09:27.750 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:09:27.867 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:09:27.867 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:09:27.870 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:09:27.870 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:09:27.870 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:09:27.871 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:09:27.871 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:27.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:27.871 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:09:27.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:27.872 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:09:27.872 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:27.872 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:27.872 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:27.873 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:27.873 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:09:27.873 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:27.873 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:27.874 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:27.874 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:27.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:27.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:27.875 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:09:27.875 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:27.875 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:27.875 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:27.876 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:27.876 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:27.877 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:27.877 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:27.877 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:27.877 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:27.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:27.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:27.878 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:09:27.878 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:27.879 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:27.879 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:27.879 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:27.879 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:27.880 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:27.880 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:27.880 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:27.880 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:27.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:27.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:27.881 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:09:27.881 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:27.881 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:27.882 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:27.882 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:27.882 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:27.882 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:27.883 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:27.883 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:27.883 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:27.883 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:27.883 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:09:27.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:27.884 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:09:27.884 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:09:27.884 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:09:27.884 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:27.885 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:09:27.889 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:09:32.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:32.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:32.891 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:09:32.951 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:09:32.952 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:09:33.061 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:09:33.062 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:09:33.062 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:09:33.062 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:09:33.062 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:09:33.063 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:09:33.063 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:33.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:33.063 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:09:33.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:33.064 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:09:33.064 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:33.064 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:33.064 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:33.065 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:33.065 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:09:33.065 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:33.065 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:33.066 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:33.066 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:33.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:33.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:33.066 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:09:33.067 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:33.067 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:33.067 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:33.067 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:33.068 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:33.068 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:33.068 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:33.068 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:33.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:33.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:33.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:33.069 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:09:33.069 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:33.070 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:33.070 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:33.070 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:33.070 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:33.070 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:33.071 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:33.071 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:33.071 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:33.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:33.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:33.072 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:09:33.072 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:33.072 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:33.072 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:33.073 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:33.073 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:33.073 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:33.073 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:33.074 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:33.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:33.075 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:33.075 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:09:33.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:33.076 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:09:33.076 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:09:33.076 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:09:33.076 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:33.077 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:09:33.078 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:09:38.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:38.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:38.080 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:09:38.123 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:09:38.124 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:09:38.242 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:09:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:09:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:09:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:09:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:09:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:09:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:09:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:09:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:09:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:09:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:09:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:09:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:09:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:09:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:09:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:09:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:38.256 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:09:38.257 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:09:43.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:43.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:43.260 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:09:43.302 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:09:43.302 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:09:43.413 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:09:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:09:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:09:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:09:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:09:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:09:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:09:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:09:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:09:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:09:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:09:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:09:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:09:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:09:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:09:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:09:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:43.427 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:09:43.428 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:09:48.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:48.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:48.431 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:09:48.474 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:09:48.475 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:09:48.587 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:09:48.587 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:09:48.587 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:09:48.588 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:09:48.588 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:09:48.588 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:09:48.588 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:48.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:48.589 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:09:48.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:48.589 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:09:48.589 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:48.590 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:48.590 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:48.590 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:48.590 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:09:48.591 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:48.591 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:48.591 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:48.591 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:48.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:48.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:48.592 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:09:48.592 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:48.592 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:48.593 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:48.593 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:48.593 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:48.593 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:48.594 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:48.594 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:48.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:48.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:48.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:48.595 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:09:48.595 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:48.595 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:48.595 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:48.596 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:48.596 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:48.596 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:48.596 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:48.596 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:48.597 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:48.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:48.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:48.597 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:09:48.598 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:48.598 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:48.598 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:48.598 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:48.599 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:48.599 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:48.599 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:48.599 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:48.599 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:48.600 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:48.600 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:09:48.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:48.600 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:09:48.601 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:09:48.601 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:09:48.601 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:48.601 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:09:48.603 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:09:53.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:53.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:53.604 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:09:53.645 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:09:53.646 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:09:53.764 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:09:53.765 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:09:53.765 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:09:53.765 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:09:53.765 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:09:53.765 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:09:53.766 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:53.766 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:53.766 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:09:53.766 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:53.767 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:09:53.767 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:53.767 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:53.767 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:53.768 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:53.768 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:09:53.768 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:53.768 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:53.768 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:53.769 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:53.769 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:53.769 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:53.769 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:09:53.770 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:53.770 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:53.770 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:53.770 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:53.771 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:53.771 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:53.771 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:53.771 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:53.771 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:53.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:53.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:53.772 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:09:53.772 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:53.773 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:53.773 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:53.773 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:53.773 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:53.773 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:53.774 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:53.774 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:53.774 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:53.774 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:53.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:53.775 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:09:53.775 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:53.776 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:53.776 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:53.776 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:53.776 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:53.777 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:53.777 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:53.777 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:53.777 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:53.778 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:53.778 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:09:53.778 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:53.778 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:09:53.779 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:09:53.779 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:09:53.779 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:53.779 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:09:53.781 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:09:58.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:58.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:09:58.784 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:09:58.827 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:09:58.827 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:09:58.940 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:09:58.941 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:09:58.941 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:09:58.941 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:09:58.942 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:09:58.942 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:09:58.942 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:58.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:58.943 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:09:58.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:58.943 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:09:58.943 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:58.943 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:58.944 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:58.944 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:58.944 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:09:58.944 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:58.945 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:58.945 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:58.945 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:58.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:58.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:58.946 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:09:58.946 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:58.946 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:58.946 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:58.947 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:58.947 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:58.947 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:58.947 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:58.948 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:58.948 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:58.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:58.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:58.949 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:09:58.949 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:58.949 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:58.949 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:58.949 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:58.950 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:58.950 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:58.950 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:58.950 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:58.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:58.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:58.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:09:58.951 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:09:58.952 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:09:58.952 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:09:58.952 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:09:58.952 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:09:58.952 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:09:58.953 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:09:58.953 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:09:58.953 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:09:58.953 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:58.954 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:58.954 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:09:58.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:09:58.954 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:09:58.954 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:09:58.955 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:09:58.955 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:09:58.955 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:09:58.956 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:10:03.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:03.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:03.958 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:04.002 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:04.003 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:04.118 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:04.118 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:04.118 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:04.119 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:04.119 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:04.119 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:04.119 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:04.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:04.120 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:04.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:04.120 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:04.120 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:04.121 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:04.121 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:04.121 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:04.121 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:04.122 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:04.122 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:04.122 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:04.122 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:04.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:04.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:04.123 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:04.123 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:04.123 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:04.124 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:04.124 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:04.124 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:04.124 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:04.124 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:04.125 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:04.125 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:04.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:04.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:04.126 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:04.126 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:04.126 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:04.126 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:04.127 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:04.127 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:04.127 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:04.127 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:04.127 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:04.128 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:04.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:04.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:04.128 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:04.129 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:04.129 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:04.129 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:04.129 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:04.129 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:04.130 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:04.130 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:04.130 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:04.130 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:04.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:04.131 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:04.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:04.131 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:04.131 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:04.132 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:04.132 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:04.132 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:04.133 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:10:09.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:09.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:09.136 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:09.180 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:09.180 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:09.297 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:09.298 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:09.298 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:09.298 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:09.299 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:09.299 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:09.299 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:09.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:09.300 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:09.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:09.300 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:09.301 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:09.301 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:09.302 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:09.302 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:09.302 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:09.302 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:09.303 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:09.303 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:09.303 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:09.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:09.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:09.304 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:09.304 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:09.304 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:09.304 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:09.305 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:09.305 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:09.305 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:09.305 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:09.305 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:09.306 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:09.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:09.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:09.306 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:09.307 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:09.307 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:09.307 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:09.307 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:09.307 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:09.308 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:09.308 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:09.308 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:09.308 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:09.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:09.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:09.309 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:09.310 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:09.310 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:09.310 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:09.310 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:09.311 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:09.311 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:09.311 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:09.312 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:09.312 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:09.312 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:09.312 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:09.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:09.313 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:09.313 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:09.313 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:09.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:09.314 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:09.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:10:14.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:14.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:14.320 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:14.361 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:14.362 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:14.469 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:14.469 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:14.469 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:14.470 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:14.470 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:14.470 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:14.470 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:14.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:14.471 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:14.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:14.471 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:14.471 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:14.472 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:14.472 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:14.472 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:14.472 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:14.473 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:14.473 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:14.473 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:14.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:14.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:14.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:14.474 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:14.474 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:14.474 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:14.475 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:14.475 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:14.475 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:14.475 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:14.476 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:14.476 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:14.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:14.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:14.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:14.477 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:14.477 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:14.477 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:14.477 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:14.478 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:14.478 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:14.478 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:14.478 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:14.479 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:14.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:14.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:14.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:14.480 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:14.480 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:14.480 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:14.480 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:14.480 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:14.481 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:14.481 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:14.481 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:14.481 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:14.482 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:14.482 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:14.482 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:14.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:14.482 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:14.483 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:14.483 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:14.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:14.483 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:14.485 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:10:19.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:19.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:19.487 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:19.538 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:19.539 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:19.653 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:19.654 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:19.654 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:19.655 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:19.655 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:19.655 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:19.656 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:19.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:19.656 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:19.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:19.657 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:19.657 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:19.657 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:19.658 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:19.658 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:19.659 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:19.660 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:19.660 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:19.660 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:19.661 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:19.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:19.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:19.662 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:19.662 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:19.662 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:19.663 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:19.663 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:19.663 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:19.664 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:19.664 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:19.664 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:19.665 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:19.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:19.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:19.667 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:19.667 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:19.667 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:19.668 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:19.668 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:19.668 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:19.668 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:19.669 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:19.669 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:19.670 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:19.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:19.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:19.671 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:19.671 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:19.671 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:19.671 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:19.672 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:19.672 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:19.672 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:19.672 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:19.673 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:19.674 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:19.674 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:19.675 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:19.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:19.675 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:19.676 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:19.677 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:19.677 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:19.678 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:19.681 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:10:24.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:24.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:24.685 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:24.728 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:24.729 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:24.844 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:24.845 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:24.845 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:24.845 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:24.845 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:24.846 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:24.846 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:24.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:24.846 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:24.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:24.847 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:24.847 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:24.847 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:24.848 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:24.848 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:24.848 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:24.848 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:24.848 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:24.849 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:24.849 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:24.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:24.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:24.850 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:24.850 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:24.850 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:24.850 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:24.850 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:24.851 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:24.851 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:24.851 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:24.851 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:24.852 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:24.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:24.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:24.852 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:24.852 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:24.853 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:24.853 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:24.853 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:24.853 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:24.854 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:24.854 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:24.854 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:24.854 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:24.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:24.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:24.855 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:24.855 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:24.855 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:24.856 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:24.856 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:24.856 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:24.856 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:24.856 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:24.857 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:24.857 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:24.857 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:24.857 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:24.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:24.858 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:24.858 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:24.858 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:24.858 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:24.859 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:24.860 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:10:29.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:29.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:29.862 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:29.905 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:29.905 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:30.018 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:30.018 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:30.019 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:30.019 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:30.019 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:30.019 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:30.020 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:30.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:30.020 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:30.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:30.020 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:30.021 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:30.021 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:30.021 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:30.021 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:30.022 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:30.022 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:30.022 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:30.022 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:30.022 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:30.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:30.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:30.023 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:30.023 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:30.024 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:30.024 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:30.025 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:30.025 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:30.026 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:30.026 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:30.026 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:30.026 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:30.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:30.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:30.027 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:30.027 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:30.028 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:30.028 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:30.028 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:30.028 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:30.029 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:30.029 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:30.029 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:30.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:30.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:30.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:30.030 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:30.030 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:30.031 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:30.031 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:30.031 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:30.031 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:30.032 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:30.032 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:30.032 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:30.032 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:30.033 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:30.033 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:30.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:30.033 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:30.034 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:30.034 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:30.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:30.034 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:30.036 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:10:35.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:35.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:35.038 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:35.081 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:35.082 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:35.197 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:35.197 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:35.197 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:35.197 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:35.198 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:35.198 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:35.198 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:35.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:35.199 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:35.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:35.199 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:35.199 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:35.200 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:35.200 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:35.200 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:35.200 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:35.201 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:35.201 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:35.201 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:35.201 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:35.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:35.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:35.202 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:35.202 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:35.202 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:35.203 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:35.203 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:35.203 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:35.203 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:35.204 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:35.204 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:35.204 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:35.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:35.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:35.205 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:35.205 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:35.205 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:35.205 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:35.205 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:35.206 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:35.206 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:35.206 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:35.206 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:35.207 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:35.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:35.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:35.207 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:35.207 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:35.208 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:35.208 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:35.208 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:35.208 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:35.209 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:35.209 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:35.209 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:35.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:35.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:35.210 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:35.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:35.210 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:35.210 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:35.211 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:35.211 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:35.211 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:35.212 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873320.827024_49.1 target= 2021-09-17T10:10:40.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:40.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:40.215 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:40.259 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:40.259 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:40.361 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:40.362 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:40.362 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:40.362 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:40.362 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:40.363 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:40.363 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:40.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:40.363 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:40.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:40.364 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:40.364 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:40.364 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:40.364 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873320.827024_49.1", 2021-09-17T10:10:40.365 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:40.365 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:40.365 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:40.365 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:08:40.827024Z", 2021-09-17T10:10:40.366 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:40.366 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:40.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:40.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:40.367 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:40.367 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:40.367 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:40.367 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:40.367 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:40.368 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:40.368 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:40.368 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:40.368 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:40.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:40.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:40.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:40.369 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:40.369 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:40.370 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:40.370 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:40.370 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:40.370 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:40.371 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:40.371 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:40.371 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:40.371 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:40.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:40.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:40.373 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:40.373 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:40.373 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:40.373 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:40.374 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:40.374 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:40.374 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:40.374 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:40.375 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:40.375 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:40.375 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:40.375 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:40.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:40.376 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:40.376 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:40.376 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:40.376 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:40.377 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:40.378 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:10:40.378 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:10:40.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:40.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:40.380 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:40.431 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:40.432 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:40.548 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:10:40.549 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:10:40.549 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:10:40.549 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:10:40.549 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:10:40.550 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:40.550 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:40.550 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:40.550 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:10:40.551 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:40.551 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:10:40.551 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:10:40.551 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:10:40.552 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873320.827024_49.1", 2021-09-17T10:10:40.552 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:10:40.552 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:10:40.552 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:10:40.553 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:08:40.827024Z", 2021-09-17T10:10:40.553 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:40.553 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:40.553 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:40.553 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:40.554 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:10:40.554 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:10:40.554 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:10:40.554 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:40.555 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:10:40.555 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:10:40.555 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:10:40.555 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:10:40.556 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:40.556 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:40.556 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:40.556 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:40.556 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:10:40.557 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:10:40.557 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:10:40.557 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:40.557 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:10:40.558 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:10:40.558 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:10:40.558 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:10:40.558 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:40.558 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:40.559 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:40.559 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:40.559 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:10:40.559 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:10:40.560 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:10:40.560 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:40.560 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:10:40.560 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:10:40.560 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:10:40.561 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:10:40.561 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:40.561 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:40.561 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:40.562 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:10:40.562 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:40.562 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:10:40.562 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:10:40.562 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:10:40.563 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:40.563 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:10:40.564 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:10:40.565 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:10:40.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:40.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:40.566 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:40.609 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:40.609 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:40.733 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:40.733 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:40.733 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:40.734 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:40.734 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:40.734 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:40.734 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:40.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:40.735 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:40.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:40.735 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:40.736 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:40.736 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:40.736 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873320.827024_49.1", 2021-09-17T10:10:40.736 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:40.736 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:40.737 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:40.737 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:08:40.827024Z", 2021-09-17T10:10:40.737 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:40.737 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:40.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:40.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:40.738 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:40.738 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:40.739 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:40.739 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:40.739 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:40.739 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:40.739 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:40.740 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:40.740 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:40.741 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:40.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:40.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:40.741 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:40.741 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:40.742 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:40.742 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:40.742 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:40.742 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:40.743 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:40.743 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:40.743 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:40.743 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:40.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:40.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:40.744 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:40.744 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:40.744 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:40.745 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:40.745 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:40.745 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:40.745 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:40.746 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:40.746 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:40.746 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:40.746 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:40.746 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:40.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:40.747 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:40.747 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:40.747 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:40.748 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:40.748 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:40.749 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:10:40.749 INFO:rgw_multi.tests:meta checkpoint 2021-09-17T10:10:40.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:40.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:40.751 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:40.791 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:40.791 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:40.877 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:10:40.904 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:10:40.905 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:40.905 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873320.827024_49.1", 2021-09-17T10:10:40.905 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:08:40.827024Z" 2021-09-17T10:10:40.905 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:40.905 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:40.906 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:40.906 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:10:40.906 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:40.906 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:40.907 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:40.907 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:10:40.907 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:40.907 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:40.908 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:40.908 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:10:40.908 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:40.908 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:10:40.909 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:10:40.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:40.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:40.913 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:41.073 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:41.073 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:41.073 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:41.074 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:41.074 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:41.074 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:41.074 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:41.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:41.075 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:41.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:41.075 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:41.076 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:41.076 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:41.076 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873320.827024_49.1", 2021-09-17T10:10:41.076 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:41.076 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:41.077 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:41.077 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:08:40.827024Z", 2021-09-17T10:10:41.077 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:41.077 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:41.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:41.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:41.078 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:41.079 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:41.079 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:41.079 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:41.079 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:41.080 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:41.080 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:41.080 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:41.080 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:41.082 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:41.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:41.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:41.083 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:41.084 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:41.084 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:41.084 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:41.085 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:41.085 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:41.085 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:41.086 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:41.086 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:41.086 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:41.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:41.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:41.087 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:41.088 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:41.088 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:41.089 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:41.089 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:41.090 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:41.090 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:41.090 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:41.091 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:41.092 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:41.092 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:41.092 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:41.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:41.093 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:41.093 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:41.094 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:41.094 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:41.095 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:41.096 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:10:41.097 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:10:41.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:41.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:41.098 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:41.139 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:41.140 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:41.255 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:10:41.256 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:10:41.256 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:10:41.256 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:10:41.256 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:10:41.256 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:41.257 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:41.257 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:41.257 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:10:41.257 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:41.258 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:10:41.258 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:10:41.258 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:10:41.258 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873320.827024_49.1", 2021-09-17T10:10:41.259 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:10:41.259 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:10:41.259 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:10:41.259 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:08:40.827024Z", 2021-09-17T10:10:41.260 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:41.260 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:41.260 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:41.260 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:41.261 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:10:41.261 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:10:41.261 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:10:41.261 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:41.262 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:10:41.262 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:10:41.262 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:10:41.262 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:10:41.262 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:41.263 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:41.263 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:41.263 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:41.263 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:10:41.264 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:10:41.264 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:10:41.264 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:41.264 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:10:41.265 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:10:41.265 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:10:41.265 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:10:41.265 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:41.265 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:41.266 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:41.266 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:41.266 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:10:41.266 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:10:41.267 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:10:41.267 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:41.267 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:10:41.267 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:10:41.267 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:10:41.268 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:10:41.268 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:41.268 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:41.268 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:41.268 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:10:41.269 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:41.269 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:10:41.269 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:10:41.269 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:10:41.270 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:41.270 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:10:41.271 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:10:41.272 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:10:41.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:41.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:41.273 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:41.316 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:41.316 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:41.419 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:41.419 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:41.420 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:41.420 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:41.421 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:41.421 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:41.421 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:41.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:41.422 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:41.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:41.423 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:41.424 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:41.424 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:41.425 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873320.827024_49.1", 2021-09-17T10:10:41.425 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:41.426 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:41.426 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:41.426 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:08:40.827024Z", 2021-09-17T10:10:41.427 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:41.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:41.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:41.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:41.429 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:41.430 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:41.430 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:41.431 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:41.431 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:41.431 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:41.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:41.432 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:41.433 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:41.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:41.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:41.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:41.435 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:41.435 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:41.435 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:41.436 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:41.436 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:41.437 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:41.437 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:41.438 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:41.438 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:41.438 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:41.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:41.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:41.440 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:41.441 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:41.441 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:41.442 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:41.442 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:41.442 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:41.442 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:41.443 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:41.443 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:41.443 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:41.443 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:41.443 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:41.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:41.444 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:41.444 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:41.444 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:41.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:41.445 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:41.446 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:10:41.446 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/git.ceph.com_ceph-c_f7edf77e75bbfd0d8d41a561025fbfc76479a1a3/qa/../src/test/rgw/rgw_multi' with args=['tests.py', 'tests_ps.py'] 2021-09-17T10:10:41.493 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-1 2021-09-17T10:10:45.283 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-2 2021-09-17T10:10:49.376 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-3 2021-09-17T10:10:49.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:49.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:49.417 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:49.473 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:49.473 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:49.558 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:10:49.573 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:10:49.574 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:49.574 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873449.418160_61.1", 2021-09-17T10:10:49.574 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:10:49.418160Z" 2021-09-17T10:10:49.574 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:49.575 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:49.575 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:49.575 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:10:49.575 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:49.576 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:49.576 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:49.576 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:10:49.576 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:49.576 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:49.577 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:49.577 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:10:49.577 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:49.577 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:10:49.578 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:10:49.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:49.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:49.580 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:49.623 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:49.623 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:49.739 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:49.739 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:49.740 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:49.740 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:49.740 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:49.741 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:49.741 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:49.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:49.742 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:49.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:49.742 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:49.743 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:49.743 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:49.744 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873445.284093_50.1", 2021-09-17T10:10:49.744 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:49.745 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:49.745 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:49.745 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:10:45.284093Z", 2021-09-17T10:10:49.746 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:49.746 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:49.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:49.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:49.748 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:49.748 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:49.749 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:49.749 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:49.750 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:49.750 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:49.751 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:49.751 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:49.752 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:49.752 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:49.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:49.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:49.753 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:49.754 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:49.754 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:49.755 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:49.755 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:49.755 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:49.756 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:49.756 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:49.757 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:49.757 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:49.757 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:49.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:49.758 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:49.759 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:49.759 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:49.760 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:49.760 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:49.761 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:49.761 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:49.761 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:49.761 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:49.762 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:49.762 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:49.762 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:49.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:49.763 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:49.763 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:49.763 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:49.763 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:49.764 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:49.765 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873449.418160_61.1 target=1_1631873445.284093_50.1 2021-09-17T10:10:54.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:54.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:54.768 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:54.811 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:54.812 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:54.923 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:54.923 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:54.923 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:54.924 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:54.924 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:54.924 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:54.924 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:54.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:54.925 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:54.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:54.925 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:54.925 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:54.926 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:54.926 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873449.418160_61.1", 2021-09-17T10:10:54.926 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:54.926 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:54.927 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:54.927 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:10:49.418160Z", 2021-09-17T10:10:54.927 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:54.927 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:54.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:54.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:54.928 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:54.928 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:54.929 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:54.929 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:54.929 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:54.929 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:54.929 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:54.930 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:54.930 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:54.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:54.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:54.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:54.931 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:54.931 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:54.931 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:54.931 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:54.932 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:54.932 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:54.932 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:54.932 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:54.933 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:54.933 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:54.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:54.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:54.933 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:54.934 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:54.934 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:54.934 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:54.934 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:54.935 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:54.935 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:54.935 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:54.935 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:54.936 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:54.936 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:54.936 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:54.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:54.936 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:54.937 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:54.937 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:54.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:54.937 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:54.939 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:10:54.939 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:10:54.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:54.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:54.941 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:54.982 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:54.983 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:55.095 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:10:55.095 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:10:55.096 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:10:55.096 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:10:55.096 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:10:55.096 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:55.097 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:55.097 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:55.097 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:10:55.097 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:55.097 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:10:55.098 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:10:55.098 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:10:55.098 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873449.418160_61.1", 2021-09-17T10:10:55.098 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:10:55.099 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:10:55.099 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:10:55.099 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:10:49.418160Z", 2021-09-17T10:10:55.099 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:55.100 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:55.100 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:55.100 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:55.100 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:10:55.100 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:10:55.101 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:10:55.101 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:55.101 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:10:55.101 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:10:55.102 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:10:55.102 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:10:55.102 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:55.102 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:55.103 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:55.103 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:55.103 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:10:55.103 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:10:55.104 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:10:55.104 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:55.105 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:10:55.105 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:10:55.105 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:10:55.106 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:10:55.106 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:55.106 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:55.106 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:55.107 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:55.107 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:10:55.107 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:10:55.108 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:10:55.108 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:55.108 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:10:55.108 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:10:55.108 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:10:55.109 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:10:55.109 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:10:55.109 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:55.109 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:55.109 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:10:55.110 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:55.110 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:10:55.110 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:10:55.110 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:10:55.111 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:55.111 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:10:55.112 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:10:55.112 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:10:55.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:55.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:55.114 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:55.158 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:55.158 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:55.275 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:55.275 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:55.275 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:55.275 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:55.276 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:55.276 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:55.276 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:55.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:55.277 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:55.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:55.277 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:55.277 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:55.277 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:55.278 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873449.418160_61.1", 2021-09-17T10:10:55.278 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:55.278 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:55.278 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:10:49.418160Z", 2021-09-17T10:10:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:55.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:55.280 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:55.280 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:55.280 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:55.280 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:55.281 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:55.281 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:55.281 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:55.281 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:55.281 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:55.282 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:55.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:55.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:55.282 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:55.283 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:55.283 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:55.283 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:55.283 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:55.284 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:55.284 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:55.284 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:55.284 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:55.284 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:55.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:55.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:55.285 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:55.285 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:55.285 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:55.286 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:55.286 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:55.286 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:55.286 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:55.286 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:55.287 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:55.287 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:55.287 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:55.287 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:55.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:55.288 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:55.288 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:55.288 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:55.289 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:55.289 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:55.290 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:10:55.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2021-09-17T10:10:55.327 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-4 2021-09-17T10:10:55.350 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-5 2021-09-17T10:10:55.396 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-6 2021-09-17T10:10:55.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:55.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:55.445 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:55.492 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:55.492 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:55.582 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:10:55.600 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:10:55.601 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:55.601 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873455.450112_73.1", 2021-09-17T10:10:55.601 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:10:55.450112Z" 2021-09-17T10:10:55.601 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:55.601 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:55.602 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:55.602 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:10:55.602 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:55.602 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:55.603 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:55.603 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:10:55.603 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:10:55.603 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:10:55.603 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:10:55.604 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:10:55.604 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:10:55.604 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:10:55.605 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:10:55.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:55.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:10:55.607 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:10:55.643 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:10:55.644 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:10:55.765 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:10:55.765 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:10:55.766 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:10:55.766 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:10:55.767 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:10:55.767 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:10:55.767 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:55.767 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:55.767 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:10:55.768 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:55.768 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:10:55.768 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:55.768 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:55.769 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873455.357132_62.1", 2021-09-17T10:10:55.769 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:55.769 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:10:55.769 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:55.770 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:10:55.357132Z", 2021-09-17T10:10:55.770 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:55.770 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:55.770 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:55.771 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:55.771 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:10:55.771 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:55.771 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:55.771 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:55.772 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:55.772 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:55.772 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:55.772 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:55.773 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:55.773 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:55.773 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:55.773 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:55.774 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:10:55.774 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:55.774 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:55.775 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:55.775 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:55.775 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:55.776 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:55.776 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:55.776 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:55.776 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:55.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:55.777 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:10:55.777 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:10:55.777 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:10:55.778 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:10:55.778 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:10:55.778 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:10:55.778 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:10:55.778 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:10:55.779 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:10:55.779 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:10:55.779 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:55.779 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:55.780 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:10:55.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:10:55.780 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:10:55.780 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:10:55.781 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:10:55.781 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:10:55.781 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:10:55.782 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873455.450112_73.1 target=1_1631873455.357132_62.1 2021-09-17T10:11:00.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:00.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:00.785 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:00.830 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:00.831 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:00.947 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:00.948 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:00.948 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:00.948 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:00.948 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:00.949 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:00.949 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:00.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:00.950 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:00.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:00.950 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:00.951 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:00.951 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:00.951 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873455.450112_73.1", 2021-09-17T10:11:00.951 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:00.951 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:00.952 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:00.952 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:10:55.450112Z", 2021-09-17T10:11:00.952 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:00.952 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:00.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:00.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:00.953 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:00.953 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:00.953 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:00.954 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:00.954 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:00.954 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:00.954 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:00.955 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:00.955 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:00.955 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:00.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:00.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:00.956 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:00.956 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:00.956 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:00.956 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:00.957 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:00.957 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:00.957 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:00.958 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:00.958 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:00.958 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:00.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:00.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:00.959 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:00.959 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:00.960 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:00.960 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:00.960 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:00.960 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:00.960 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:00.961 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:00.961 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:00.961 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:00.961 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:00.962 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:00.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:00.962 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:00.962 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:00.962 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:00.963 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:00.963 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:00.964 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:11:00.965 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:11:00.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:00.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:00.966 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:01.009 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:01.010 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:01.118 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:11:01.118 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:11:01.118 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:11:01.119 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:11:01.119 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:11:01.119 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:01.119 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:01.119 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.120 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:11:01.120 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:01.120 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:11:01.120 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:01.121 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:01.122 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873455.450112_73.1", 2021-09-17T10:11:01.122 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:01.122 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:11:01.122 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:01.123 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:10:55.450112Z", 2021-09-17T10:11:01.123 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:01.123 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.123 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.124 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:01.124 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:11:01.124 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:01.124 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:01.125 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:01.125 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:01.125 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:01.125 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:01.125 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:01.126 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:01.126 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.126 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.126 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:01.127 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:11:01.127 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:01.127 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:01.127 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:01.127 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:01.128 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:01.128 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:01.128 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:01.128 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:01.129 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.129 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.129 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:01.129 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:11:01.129 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:01.130 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:01.130 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:01.130 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:01.130 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:01.131 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:01.131 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:01.131 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:01.131 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.132 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.132 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:11:01.132 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.132 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:11:01.132 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:11:01.133 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:11:01.133 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.133 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:11:01.134 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:11:01.135 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:11:01.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:01.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:01.137 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:01.182 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:01.182 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:01.301 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:01.301 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:01.302 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:01.302 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:01.303 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:01.303 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:01.303 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:01.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:01.304 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:01.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:01.305 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:01.305 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:01.306 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:01.306 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873455.450112_73.1", 2021-09-17T10:11:01.307 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:01.307 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:01.308 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:01.308 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:10:55.450112Z", 2021-09-17T10:11:01.308 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:01.308 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:01.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:01.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:01.310 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:01.310 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:01.310 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:01.311 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:01.311 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:01.311 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:01.311 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:01.311 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:01.312 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:01.312 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:01.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:01.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:01.313 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:01.313 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:01.313 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:01.314 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:01.314 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:01.314 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:01.314 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:01.314 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:01.315 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:01.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:01.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:01.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:01.316 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:01.316 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:01.316 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:01.317 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:01.317 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:01.317 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:01.317 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:01.317 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:01.318 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:01.318 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:01.318 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:01.318 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:01.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:01.319 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:01.319 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:01.319 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:01.319 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:01.320 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:01.321 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:11:01.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:01.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:01.468 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:01.507 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:01.507 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:01.595 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:11:01.624 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:11:01.625 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:01.625 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873455.450112_73.1", 2021-09-17T10:11:01.625 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:10:55.450112Z" 2021-09-17T10:11:01.625 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.626 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:01.626 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:01.626 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:01.626 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.627 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:01.627 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:01.627 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:01.627 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.628 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:01.628 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:01.628 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:01.628 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.629 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:11:01.630 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:11:01.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:01.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:01.632 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:01.677 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:01.677 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:01.788 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:01.789 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:01.789 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:01.789 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:01.790 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:01.790 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:01.790 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:01.790 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:01.790 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:01.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:01.791 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:01.791 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:01.791 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:01.792 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873455.450112_73.1", 2021-09-17T10:11:01.792 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:01.792 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:01.792 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:01.793 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:10:55.450112Z", 2021-09-17T10:11:01.793 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:01.793 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:01.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:01.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:01.794 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:01.794 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:01.794 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:01.794 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:01.795 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:01.795 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:01.795 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:01.795 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:01.795 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:01.796 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:01.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:01.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:01.796 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:01.797 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:01.797 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:01.797 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:01.797 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:01.797 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:01.798 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:01.798 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:01.798 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:01.798 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:01.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:01.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:01.799 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:01.799 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:01.799 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:01.800 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:01.800 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:01.800 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:01.800 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:01.801 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:01.801 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:01.801 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:01.801 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:01.801 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:01.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:01.802 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:01.802 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:01.802 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:01.803 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:01.803 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:01.804 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:11:01.805 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:11:01.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:01.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:01.806 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:01.851 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:01.851 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:01.967 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:11:01.967 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:11:01.967 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:11:01.968 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:11:01.968 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:11:01.968 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:01.968 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:01.969 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.969 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:11:01.969 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:01.969 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:11:01.969 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:01.970 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:01.970 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873455.450112_73.1", 2021-09-17T10:11:01.970 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:01.970 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:11:01.971 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:01.971 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:10:55.450112Z", 2021-09-17T10:11:01.971 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:01.971 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.971 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.972 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:01.972 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:11:01.972 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:01.972 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:01.973 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:01.973 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:01.973 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:01.973 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:01.973 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:01.974 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:01.974 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.974 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.974 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:01.975 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:11:01.975 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:01.975 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:01.975 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:01.975 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:01.976 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:01.976 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:01.976 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:01.976 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:01.977 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.977 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.977 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:01.977 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:11:01.977 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:01.978 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:01.978 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:01.978 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:01.978 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:01.979 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:01.979 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:01.979 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:01.979 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.979 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.980 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:11:01.980 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:01.980 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:11:01.980 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:11:01.981 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:11:01.981 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:01.981 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:11:01.983 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:11:01.983 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:11:01.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:01.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:01.985 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:02.028 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:02.028 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:02.141 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:02.141 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:02.142 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:02.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873455.450112_73.1", 2021-09-17T10:11:02.144 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:10:55.450112Z", 2021-09-17T10:11:02.145 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:02.146 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:02.147 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:02.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:02.149 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:02.150 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:02.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:02.152 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:02.153 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:02.153 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:02.153 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:02.153 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:02.155 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:02.155 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:02.157 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:11:02.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2021-09-17T10:11:02.193 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-7 2021-09-17T10:11:02.219 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-8 2021-09-17T10:11:02.261 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-9 2021-09-17T10:11:02.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:02.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:02.313 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:02.352 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:02.352 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:02.436 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:11:02.459 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:11:02.460 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:02.460 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873462.313467_85.1", 2021-09-17T10:11:02.460 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:02.313467Z" 2021-09-17T10:11:02.460 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:02.461 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:02.461 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:02.461 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:02.461 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:02.462 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:02.462 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:02.462 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:02.462 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:02.463 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:02.463 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:02.463 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:02.463 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:02.463 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:11:02.464 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:11:02.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:02.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:02.466 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:02.519 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:02.520 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:02.656 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:02.656 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:02.656 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:02.657 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:02.657 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:02.657 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:02.657 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:02.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:02.658 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:02.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:02.658 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:02.659 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:02.659 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:02.659 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873462.229713_76.1", 2021-09-17T10:11:02.659 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:02.659 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:02.660 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:02.660 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:02.229713Z", 2021-09-17T10:11:02.660 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:02.660 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:02.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:02.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:02.661 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:02.661 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:02.662 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:02.662 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:02.662 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:02.662 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:02.662 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:02.663 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:02.663 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:02.663 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:02.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:02.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:02.664 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:02.664 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:02.664 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:02.664 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:02.665 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:02.665 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:02.665 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:02.665 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:02.666 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:02.666 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:02.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:02.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:02.667 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:02.667 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:02.667 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:02.667 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:02.667 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:02.668 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:02.668 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:02.668 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:02.668 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:02.669 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:02.669 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:02.669 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:02.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:02.669 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:02.670 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:02.670 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:02.670 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:02.670 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:02.672 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873462.313467_85.1 target=1_1631873462.229713_76.1 2021-09-17T10:11:07.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:07.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:07.674 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:07.717 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:07.717 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:07.828 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:07.829 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:07.829 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:07.829 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:07.830 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:07.830 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:07.830 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:07.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:07.831 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:07.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:07.831 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:07.831 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:07.832 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:07.832 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873462.313467_85.1", 2021-09-17T10:11:07.832 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:07.832 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:07.832 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:07.833 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:02.313467Z", 2021-09-17T10:11:07.833 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:07.833 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:07.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:07.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:07.834 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:07.835 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:07.835 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:07.835 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:07.836 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:07.836 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:07.836 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:07.836 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:07.836 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:07.837 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:07.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:07.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:07.837 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:07.838 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:07.838 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:07.838 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:07.838 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:07.838 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:07.839 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:07.839 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:07.839 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:07.839 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:07.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:07.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:07.840 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:07.840 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:07.840 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:07.841 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:07.841 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:07.841 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:07.841 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:07.842 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:07.842 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:07.843 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:07.843 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:07.844 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:07.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:07.844 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:07.845 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:07.845 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:07.845 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:07.846 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:07.847 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:11:07.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:11:07.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:07.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:07.849 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:07.891 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:07.892 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:08.003 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:11:08.004 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:11:08.004 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:11:08.004 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:11:08.004 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:11:08.005 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:08.005 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:08.005 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:08.005 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:11:08.005 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:08.006 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:11:08.006 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:08.006 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:08.006 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873462.313467_85.1", 2021-09-17T10:11:08.007 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:08.007 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:11:08.007 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:08.007 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:02.313467Z", 2021-09-17T10:11:08.008 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:08.008 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:08.008 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:08.008 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:08.008 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:11:08.009 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:08.009 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:08.009 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:08.009 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:08.010 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:08.010 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:08.010 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:08.010 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:08.011 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:08.011 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:08.011 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:08.011 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:11:08.011 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:08.012 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:08.012 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:08.012 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:08.012 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:08.012 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:08.013 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:08.013 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:08.013 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:08.013 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:08.014 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:08.014 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:11:08.014 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:08.014 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:08.014 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:08.015 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:08.015 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:08.015 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:08.015 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:08.016 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:08.016 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:08.016 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:08.016 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:11:08.016 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:08.017 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:11:08.017 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:11:08.017 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:11:08.017 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:08.018 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:11:08.019 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:11:08.019 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:11:08.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:08.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:08.021 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:08.064 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:08.065 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:08.184 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:08.185 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:08.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:08.187 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873462.313467_85.1", 2021-09-17T10:11:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:08.188 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:02.313467Z", 2021-09-17T10:11:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:08.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:08.190 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:08.191 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:08.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:08.193 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:08.194 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:08.195 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:08.196 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:08.197 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:08.198 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:08.199 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:08.199 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:08.200 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:11:08.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:08.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:08.491 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:08.528 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:08.528 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:08.613 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:11:08.637 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:11:08.638 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:08.638 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:08.639 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:11:08.639 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:08.640 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:08.640 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:08.640 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:08.641 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:08.641 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:08.641 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:08.642 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:08.642 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:08.642 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:08.643 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:08.643 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:08.644 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:08.644 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:11:08.646 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:11:08.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:08.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:08.649 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:08.691 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:08.692 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:08.808 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:08.808 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:08.808 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:08.809 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:08.809 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:08.809 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:08.809 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:08.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:08.810 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:08.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:08.810 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:08.811 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:08.811 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:08.811 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.285571_86.1", 2021-09-17T10:11:08.811 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:08.812 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:08.812 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:08.812 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.285571Z", 2021-09-17T10:11:08.812 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:08.812 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:08.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:08.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:08.813 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:08.813 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:08.814 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:08.814 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:08.814 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:08.814 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:08.814 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:08.815 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:08.815 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:08.815 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:08.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:08.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:08.816 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:08.816 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:08.816 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:08.817 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:08.817 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:08.817 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:08.817 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:08.817 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:08.818 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:08.818 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:08.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:08.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:08.819 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:08.819 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:08.819 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:08.819 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:08.819 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:08.820 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:08.820 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:08.820 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:08.820 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:08.821 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:08.821 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:08.821 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:08.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:08.821 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:08.822 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:08.822 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:08.822 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:08.822 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:08.824 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631873468.462214_91.1 target=1_1631873468.285571_86.1 2021-09-17T10:11:13.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:13.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:13.827 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:13.872 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:13.988 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:13.989 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:13.989 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:13.989 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:13.989 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:13.990 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:13.990 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:13.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:13.990 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:13.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:13.991 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:13.991 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:13.991 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:13.991 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:13.992 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:13.992 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:13.992 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:13.992 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:13.993 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:13.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:13.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:13.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:13.993 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:13.994 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:13.994 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:13.994 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:13.994 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:13.995 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:13.995 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:13.995 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:13.995 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:13.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:13.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:13.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:13.996 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:13.996 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:13.997 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:13.997 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:13.997 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:13.997 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:13.997 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:13.998 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:13.998 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:13.998 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:13.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:13.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:13.999 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:13.999 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:13.999 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:13.999 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:14.000 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:14.000 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:14.000 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:14.000 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:14.001 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:14.001 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:14.001 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:14.001 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:14.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:14.002 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:14.002 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:14.002 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:14.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:14.002 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:14.004 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:11:14.004 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:11:14.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:14.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:14.006 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:14.048 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:14.048 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:14.152 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:11:14.153 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:11:14.153 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:11:14.153 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:11:14.154 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:11:14.154 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:14.154 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:14.154 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:14.155 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:11:14.155 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:14.155 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:11:14.155 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:14.156 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:14.156 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:14.156 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:14.156 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:11:14.157 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:14.157 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:14.157 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:14.157 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:14.158 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:14.158 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:14.158 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:11:14.158 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:14.159 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:14.159 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:14.159 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:14.159 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:14.159 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:14.160 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:14.160 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:14.160 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:14.160 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:14.161 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:14.161 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:11:14.161 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:14.161 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:14.162 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:14.162 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:14.162 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:14.162 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:14.162 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:14.163 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:14.163 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:14.163 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:14.163 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:14.164 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:11:14.164 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:14.164 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:14.164 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:14.164 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:14.165 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:14.165 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:14.165 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:14.165 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:14.165 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:14.166 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:14.166 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:11:14.166 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:14.166 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:11:14.167 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:11:14.167 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:11:14.167 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:14.167 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:11:14.169 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:11:14.169 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:11:14.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:14.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:14.171 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:14.212 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:14.213 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:14.324 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:14.324 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:14.325 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:14.325 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:14.325 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:14.325 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:14.325 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:14.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:14.326 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:14.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:14.326 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:14.327 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:14.327 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:14.327 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:14.327 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:14.327 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:14.328 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:14.328 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:14.328 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:14.328 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:14.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:14.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:14.329 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:14.329 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:14.329 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:14.330 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:14.330 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:14.330 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:14.330 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:14.331 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:14.331 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:14.331 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:14.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:14.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:14.332 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:14.332 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:14.332 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:14.332 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:14.332 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:14.333 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:14.333 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:14.333 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:14.333 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:14.334 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:14.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:14.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:14.334 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:14.334 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:14.335 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:14.336 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:14.337 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:14.338 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:14.338 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:14.339 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:11:14.352 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2021-09-17T10:11:14.353 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-10 2021-09-17T10:11:14.379 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-11 2021-09-17T10:11:14.428 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-12 2021-09-17T10:11:15.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:15.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:15.080 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:15.126 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:15.126 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:15.204 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:11:15.235 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:11:15.235 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:15.235 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:15.236 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:11:15.236 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:15.236 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:15.236 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873474.474631_67.1", 2021-09-17T10:11:15.237 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:14.474631Z" 2021-09-17T10:11:15.237 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:15.237 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:15.237 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:15.237 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:15.238 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:15.238 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:15.238 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:15.238 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:15.239 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:15.239 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:11:15.240 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:11:15.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:15.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:15.242 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:15.282 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:15.283 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:15.377 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:15.377 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:15.377 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:15.377 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:15.378 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:15.378 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:15.378 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:15.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:15.379 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:15.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:15.379 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:15.379 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:15.380 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:15.380 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:15.380 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:15.381 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:15.381 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:15.381 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:15.381 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:15.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:15.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:15.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:15.382 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:15.382 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:15.383 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:15.383 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873474.387990_56.1", 2021-09-17T10:11:15.383 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:15.383 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:15.384 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:15.384 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:14.387990Z", 2021-09-17T10:11:15.384 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:15.384 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:15.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:15.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:15.385 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:15.385 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:15.385 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:15.386 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:15.386 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:15.386 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:15.386 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:15.386 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:15.387 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:15.387 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:15.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:15.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:15.388 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:15.389 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:15.390 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:15.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:15.391 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:15.393 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631873474.474631_67.1 target=1_1631873474.387990_56.1 2021-09-17T10:11:20.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:20.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:20.395 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:20.437 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:20.437 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:20.550 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:20.550 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:20.551 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:20.551 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:20.551 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:20.551 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:20.552 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:20.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:20.552 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:20.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:20.552 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:20.553 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:20.553 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:20.553 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:20.553 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:20.554 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:20.554 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:20.554 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:20.554 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:20.555 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:20.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:20.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:20.555 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:20.555 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:20.556 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:20.556 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873474.474631_67.1", 2021-09-17T10:11:20.556 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:20.556 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:20.557 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:20.557 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:14.474631Z", 2021-09-17T10:11:20.557 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:20.557 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:20.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:20.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:20.558 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:20.558 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:20.558 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:20.559 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:20.559 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:20.559 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:20.559 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:20.560 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:20.560 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:20.560 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:20.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:20.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:20.561 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:20.561 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:20.561 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:20.561 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:20.562 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:20.562 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:20.562 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:20.562 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:20.562 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:20.563 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:20.563 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:20.563 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:20.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:20.564 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:20.564 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:20.564 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:20.564 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:20.565 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:20.566 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:11:20.566 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:11:20.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:20.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:20.568 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:20.611 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:20.611 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:20.721 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:11:20.721 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:11:20.721 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:11:20.722 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:11:20.722 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:11:20.722 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:20.722 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:20.723 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:20.723 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:11:20.723 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:20.723 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:11:20.723 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:20.724 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:20.724 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:20.724 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:20.724 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:11:20.725 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:20.725 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:20.725 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:20.725 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:20.725 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:20.726 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:20.726 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:11:20.726 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:20.726 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:20.727 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873474.474631_67.1", 2021-09-17T10:11:20.727 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:20.727 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:20.727 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:20.727 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:14.474631Z", 2021-09-17T10:11:20.728 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:20.728 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:20.728 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:20.728 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:20.729 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:11:20.729 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:20.729 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:20.729 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:20.729 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:20.730 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:20.730 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:20.730 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:20.731 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:20.731 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:20.731 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:20.731 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:20.732 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:11:20.732 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:20.732 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:20.733 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:20.733 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:20.733 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:20.733 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:20.733 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:20.734 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:20.734 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:20.734 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:20.734 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:11:20.735 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:20.735 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:11:20.735 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:11:20.735 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:11:20.735 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:20.736 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:11:20.737 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:11:20.738 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:11:20.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:20.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:20.739 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:20.782 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:20.782 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:20.896 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:20.897 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:20.897 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:20.897 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:20.898 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:20.898 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:20.898 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:20.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:20.899 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:20.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:20.899 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:20.899 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:20.899 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:20.900 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:20.900 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:20.900 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:20.900 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:20.901 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:20.901 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:20.901 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:20.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:20.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:20.902 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:20.902 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:20.902 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:20.902 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873474.474631_67.1", 2021-09-17T10:11:20.903 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:20.903 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:20.903 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:20.903 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:14.474631Z", 2021-09-17T10:11:20.903 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:20.904 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:20.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:20.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:20.904 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:20.905 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:20.905 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:20.905 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:20.905 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:20.905 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:20.906 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:20.906 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:20.906 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:20.906 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:20.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:20.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:20.907 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:20.907 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:20.907 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:20.908 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:20.908 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:20.908 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:20.908 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:20.909 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:20.909 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:20.909 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:20.909 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:20.909 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:20.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:20.910 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:20.910 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:20.910 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:20.910 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:20.911 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:20.912 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:11:20.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:20.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:20.914 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:20.963 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:20.963 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:21.010 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:21.030+0000 7f2238c30b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:21.010 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:21.030+0000 7f2238c30b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:21.011 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:21.030+0000 7f2238c30b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:11:21.011 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:21.030+0000 7f2238c30b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:21.067 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:21.086+0000 7f2238c30b00 1 bucket sync caught up with source: 2021-09-17T10:11:21.068 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000001.3.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.5.5, , ] 2021-09-17T10:11:21.068 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000001.3.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.5.5, , ] 2021-09-17T10:11:21.068 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:21.086+0000 7f2238c30b00 0 bucket checkpoint complete 2021-09-17T10:11:21.097 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-10 zones={test-zone2, test-zone1} 2021-09-17T10:11:21.106 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:21.173 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T10:11:21.173 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T10:11:21.174 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T10:11:21.175 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:21.175 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:21.182 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T10:11:21.182 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T10:11:21.183 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T10:11:21.183 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:21.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:21.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:21.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:21.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:21.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:21.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:21.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:21.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:21.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:22.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:22.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:22.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:22.350 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-10 zones={test-zone2, test-zone1} 2021-09-17T10:11:22.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:22.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:22.352 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:22.398 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:22.398 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:22.447 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:22.464+0000 7f21a97f6b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:22.447 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:22.464+0000 7f21a97f6b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:22.448 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:22.464+0000 7f21a97f6b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:11:22.448 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:22.464+0000 7f21a97f6b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:22.499 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:22.516+0000 7f21a97f6b00 1 bucket sync caught up with source: 2021-09-17T10:11:22.499 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000001.3.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.5.5, , ] 2021-09-17T10:11:22.500 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000001.3.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.5.5, , ] 2021-09-17T10:11:22.500 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:22.516+0000 7f21a97f6b00 0 bucket checkpoint complete 2021-09-17T10:11:22.525 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-10 zones={test-zone3, test-zone1} 2021-09-17T10:11:22.534 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:22.587 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T10:11:22.588 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T10:11:22.588 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T10:11:22.589 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:22.589 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:22.596 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T10:11:22.596 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T10:11:22.596 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T10:11:22.597 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:22.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:22.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:22.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:22.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:22.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:23.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:23.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:23.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:23.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:23.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:23.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:23.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:23.749 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-10 zones={test-zone3, test-zone1} 2021-09-17T10:11:23.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:23.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:23.751 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:23.854 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:23.874+0000 7fc2d787eb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:23.854 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:23.874+0000 7fc2d787eb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:23.854 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:23.874+0000 7fc2d787eb00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:11:23.855 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:23.874+0000 7fc2d787eb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:23.909 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:23.930+0000 7fc2d787eb00 1 bucket sync caught up with source: 2021-09-17T10:11:23.910 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000001.3.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.5.5, , ] 2021-09-17T10:11:23.910 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000001.3.5, , 00000000001.4.5, , , , 00000000001.9.5, 00000000001.5.5, , ] 2021-09-17T10:11:23.911 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:23.930+0000 7fc2d787eb00 0 bucket checkpoint complete 2021-09-17T10:11:23.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:23.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:23.939 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:23.988 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:23.988 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:24.037 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:24.056+0000 7fb12e8b3b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:24.037 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:24.056+0000 7fb12e8b3b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:24.038 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:24.056+0000 7fb12e8b3b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:11:24.038 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:24.056+0000 7fb12e8b3b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:24.092 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:24.108+0000 7fb12e8b3b00 1 bucket sync caught up with source: 2021-09-17T10:11:24.092 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000001.73.5, , 00000000001.22.5, , , , 00000000001.53.5, 00000000001.34.5, , ] 2021-09-17T10:11:24.093 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000001.73.5, , 00000000001.22.5, , , , 00000000001.53.5, 00000000001.34.5, , ] 2021-09-17T10:11:24.093 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:24.112+0000 7fb12e8b3b00 0 bucket checkpoint complete 2021-09-17T10:11:24.115 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-11 zones={test-zone1, test-zone2} 2021-09-17T10:11:24.124 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:24.172 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T10:11:24.173 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T10:11:24.173 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T10:11:24.174 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:24.174 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:24.181 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T10:11:24.181 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T10:11:24.182 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T10:11:24.182 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:24.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:24.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:24.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:24.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:24.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:24.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:24.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:24.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:24.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:25.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:25.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:25.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:25.321 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-11 zones={test-zone1, test-zone2} 2021-09-17T10:11:25.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:25.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:25.323 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:25.373 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:25.374 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:25.421 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:25.440+0000 7f2052f33b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:25.422 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:25.440+0000 7f2052f33b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:25.422 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:25.440+0000 7f2052f33b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:11:25.422 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:25.440+0000 7f2052f33b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:25.477 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:25.496+0000 7f2052f33b00 1 bucket sync caught up with source: 2021-09-17T10:11:25.478 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000001.73.5, , 00000000001.22.5, , , , 00000000001.53.5, 00000000001.34.5, , ] 2021-09-17T10:11:25.478 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000001.73.5, , 00000000001.22.5, , , , 00000000001.53.5, 00000000001.34.5, , ] 2021-09-17T10:11:25.478 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:25.496+0000 7f2052f33b00 0 bucket checkpoint complete 2021-09-17T10:11:25.507 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-11 zones={test-zone3, test-zone2} 2021-09-17T10:11:25.516 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:25.563 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T10:11:25.564 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T10:11:25.564 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T10:11:25.565 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:25.565 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:25.575 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T10:11:25.576 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T10:11:25.577 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T10:11:25.578 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:25.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:25.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:25.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:25.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:25.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:26.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:26.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:26.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:26.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:26.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:26.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:26.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:26.718 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-11 zones={test-zone3, test-zone2} 2021-09-17T10:11:26.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:26.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:26.719 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:26.771 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:26.771 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:26.818 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:26.836+0000 7f0787639b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:26.819 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:26.836+0000 7f0787639b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:26.819 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:26.836+0000 7f0787639b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:11:26.819 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:26.836+0000 7f0787639b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:26.873 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:26.892+0000 7f0787639b00 1 bucket sync caught up with source: 2021-09-17T10:11:26.873 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000001.103.5, , 00000000001.54.5, , , , 00000000001.108.5, 00000000001.52.5, , ] 2021-09-17T10:11:26.874 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000001.103.5, , 00000000001.54.5, , , , 00000000001.108.5, 00000000001.52.5, , ] 2021-09-17T10:11:26.874 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:26.892+0000 7f0787639b00 0 bucket checkpoint complete 2021-09-17T10:11:26.902 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-12 zones={test-zone1, test-zone3} 2021-09-17T10:11:26.911 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:26.959 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T10:11:26.960 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T10:11:26.960 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T10:11:26.960 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:26.961 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:26.967 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T10:11:26.968 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T10:11:26.968 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T10:11:26.969 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:27.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:27.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:27.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:27.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:27.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:27.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:27.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:27.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:27.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:27.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:27.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:28.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:28.138 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-12 zones={test-zone1, test-zone3} 2021-09-17T10:11:28.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:28.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:28.140 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:28.190 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:28.190 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:28.236 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:28.254+0000 7fd56c136b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:28.236 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:28.254+0000 7fd56c136b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:28.237 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:28.254+0000 7fd56c136b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:11:28.237 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:28.254+0000 7fd56c136b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:28.291 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:28.310+0000 7fd56c136b00 1 bucket sync caught up with source: 2021-09-17T10:11:28.292 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000001.103.5, , 00000000001.54.5, , , , 00000000001.108.5, 00000000001.52.5, , ] 2021-09-17T10:11:28.292 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000001.103.5, , 00000000001.54.5, , , , 00000000001.108.5, 00000000001.52.5, , ] 2021-09-17T10:11:28.292 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:28.310+0000 7fd56c136b00 0 bucket checkpoint complete 2021-09-17T10:11:28.316 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-12 zones={test-zone2, test-zone3} 2021-09-17T10:11:28.325 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:28.375 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T10:11:28.376 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T10:11:28.376 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T10:11:28.377 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:28.377 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:28.384 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-17T10:11:28.384 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-17T10:11:28.385 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-17T10:11:28.385 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:28.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:28.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:28.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-17T10:11:28.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:28.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:28.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-17T10:11:28.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:29.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:29.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-17T10:11:29.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:29.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:29.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:29.546 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-12 zones={test-zone2, test-zone3} 2021-09-17T10:11:29.546 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2021-09-17T10:11:29.547 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-13 2021-09-17T10:11:29.573 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-14 2021-09-17T10:11:29.622 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-15 2021-09-17T10:11:29.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:29.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:29.820 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:29.822 INFO:tasks.rgw.c2.client.0.smithi050.stdout:2021-09-17T10:11:29.838+0000 7fd538ff9700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T10:11:29.873 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:29.873 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:29.978 INFO:tasks.rgw.c2.client.0.smithi050.stdout:2021-09-17T10:11:29.998+0000 7fd52eff5700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T10:11:30.009 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:11:30.023 INFO:tasks.rgw.c2.client.1.smithi050.stdout:2021-09-17T10:11:30.042+0000 7f0ae0ff9700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T10:11:30.047 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:11:30.047 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:30.047 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:30.048 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:11:30.048 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:30.048 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:30.048 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873489.675557_94.1", 2021-09-17T10:11:30.049 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:29.675557Z" 2021-09-17T10:11:30.049 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:30.049 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:30.049 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:30.050 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:30.050 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:30.050 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:30.051 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:30.051 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:30.051 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:30.051 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:11:30.052 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:11:30.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:30.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:30.054 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:30.108 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:30.109 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:30.204 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:30.204 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:30.205 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:30.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:30.207 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:30.208 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:30.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:30.210 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873489.663639_91.1", 2021-09-17T10:11:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:29.663639Z", 2021-09-17T10:11:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:30.215 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:30.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:30.217 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:30.218 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:30.219 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:30.220 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:30.221 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:30.222 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:30.222 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:30.224 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631873489.675557_94.1 target=1_1631873489.663639_91.1 2021-09-17T10:11:35.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:35.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:35.226 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:35.270 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:35.271 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:35.381 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:35.381 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:35.381 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:35.381 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:35.382 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:35.382 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:35.382 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:35.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:35.383 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:35.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:35.383 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:35.383 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:35.383 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:35.384 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:35.384 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:35.384 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:35.384 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:35.385 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:35.385 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:35.385 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:35.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:35.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:35.386 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:35.386 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:35.386 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:35.387 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873489.675557_94.1", 2021-09-17T10:11:35.387 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:35.387 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:35.387 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:35.388 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:29.675557Z", 2021-09-17T10:11:35.388 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:35.388 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:35.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:35.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:35.389 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:35.389 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:35.389 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:35.390 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:35.390 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:35.390 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:35.390 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:35.391 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:35.391 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:35.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:35.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:35.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:35.392 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:35.392 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:35.392 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:35.392 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:35.393 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:35.393 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:35.393 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:35.393 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:35.394 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:35.394 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:35.394 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:35.394 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:35.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:35.395 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:35.395 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:35.395 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:35.395 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:35.396 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:35.397 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:11:35.397 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:11:35.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:35.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:35.399 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:35.443 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:35.443 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:35.554 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:11:35.555 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:11:35.555 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:11:35.555 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:11:35.555 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:11:35.556 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:35.556 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:35.556 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:35.556 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:11:35.557 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:35.557 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:11:35.557 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:35.557 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:35.557 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:35.558 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:35.558 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:11:35.558 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:35.558 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:35.559 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:35.559 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:35.559 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:35.559 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:35.559 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:11:35.560 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:35.560 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:35.560 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873489.675557_94.1", 2021-09-17T10:11:35.560 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:35.560 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:35.561 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:35.561 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:29.675557Z", 2021-09-17T10:11:35.561 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:35.561 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:35.562 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:35.562 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:35.562 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:11:35.562 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:35.563 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:35.563 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:35.563 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:35.563 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:35.563 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:35.564 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:35.564 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:35.564 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:35.564 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:35.565 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:35.565 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:11:35.565 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:35.565 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:35.565 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:35.566 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:35.566 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:35.566 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:35.566 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:35.567 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:35.567 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:35.567 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:35.567 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:11:35.567 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:35.568 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:11:35.568 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:11:35.568 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:11:35.568 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:35.569 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:11:35.570 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:11:35.570 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:11:35.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:35.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:35.572 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:35.615 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:35.616 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:35.731 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:35.731 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:35.732 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:35.732 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:35.732 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:35.732 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:35.733 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:35.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:35.733 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:35.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:35.734 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:35.734 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:35.734 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:35.734 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:35.734 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:35.735 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:35.736 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:35.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:35.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:35.736 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873489.675557_94.1", 2021-09-17T10:11:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:35.737 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:35.738 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:35.738 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:29.675557Z", 2021-09-17T10:11:35.738 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:35.739 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:35.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:35.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:35.739 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:35.739 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:35.740 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:35.740 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:35.740 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:35.740 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:35.741 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:35.741 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:35.741 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:35.741 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:35.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:35.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:35.742 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:35.742 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:35.742 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:35.743 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:35.743 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:35.743 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:35.743 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:35.744 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:35.744 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:35.744 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:35.744 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:35.745 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:35.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:35.745 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:35.745 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:35.746 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:35.746 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:35.746 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:35.747 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:11:35.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:35.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:35.749 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:35.798 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:35.798 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:35.866+0000 7f8dc6f04b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:35.866+0000 7f8dc6f04b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:35.866+0000 7f8dc6f04b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:11:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:35.866+0000 7f8dc6f04b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:35.903 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:35.922+0000 7f8dc6f04b00 1 bucket sync caught up with source: 2021-09-17T10:11:35.903 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2021-09-17T10:11:35.903 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2021-09-17T10:11:35.904 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:35.922+0000 7f8dc6f04b00 0 bucket checkpoint complete 2021-09-17T10:11:35.933 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-13 zones={test-zone2, test-zone1} 2021-09-17T10:11:35.943 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:35.995 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:35.995 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:36.002 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:36.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:36.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:36.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:36.338 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-13 zones={test-zone2, test-zone1} 2021-09-17T10:11:36.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:36.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:36.340 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:36.388 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:36.389 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:36.438 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:36.456+0000 7efc0257fb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:36.438 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:36.456+0000 7efc0257fb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:36.439 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:36.456+0000 7efc0257fb00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:11:36.439 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:36.456+0000 7efc0257fb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:36.490 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:36.508+0000 7efc0257fb00 1 bucket sync caught up with source: 2021-09-17T10:11:36.491 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2021-09-17T10:11:36.491 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2021-09-17T10:11:36.491 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:36.508+0000 7efc0257fb00 0 bucket checkpoint complete 2021-09-17T10:11:36.517 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-13 zones={test-zone3, test-zone1} 2021-09-17T10:11:36.525 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:36.571 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:36.572 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:36.578 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:36.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:36.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:36.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:36.947 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-13 zones={test-zone3, test-zone1} 2021-09-17T10:11:36.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:36.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:36.949 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:36.999 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:37.049 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:37.070+0000 7fee2ac9fb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:37.050 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:37.070+0000 7fee2ac9fb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:37.050 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:37.070+0000 7fee2ac9fb00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:11:37.050 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:37.070+0000 7fee2ac9fb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:37.107 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:37.126+0000 7fee2ac9fb00 1 bucket sync caught up with source: 2021-09-17T10:11:37.107 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2021-09-17T10:11:37.107 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2021-09-17T10:11:37.108 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:37.126+0000 7fee2ac9fb00 0 bucket checkpoint complete 2021-09-17T10:11:37.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:37.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:37.137 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:37.188 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:37.189 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:37.234 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:37.252+0000 7fcda3f2cb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:37.235 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:37.252+0000 7fcda3f2cb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:37.235 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:37.252+0000 7fcda3f2cb00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:11:37.235 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:37.252+0000 7fcda3f2cb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:37.289 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:37.308+0000 7fcda3f2cb00 1 bucket sync caught up with source: 2021-09-17T10:11:37.290 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000001.84.5, , , , , , , , , ] 2021-09-17T10:11:37.290 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000001.84.5, , , , , , , , , ] 2021-09-17T10:11:37.290 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:37.308+0000 7fcda3f2cb00 0 bucket checkpoint complete 2021-09-17T10:11:37.320 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-14 zones={test-zone1, test-zone2} 2021-09-17T10:11:37.330 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:37.379 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:37.380 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:37.386 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:37.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:37.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:37.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:37.726 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-14 zones={test-zone1, test-zone2} 2021-09-17T10:11:37.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:37.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:37.728 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:37.777 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:37.778 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:37.826 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:37.844+0000 7f10e2b82b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:37.826 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:37.844+0000 7f10e2b82b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:37.826 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:37.844+0000 7f10e2b82b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:11:37.827 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:37.844+0000 7f10e2b82b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:37.879 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:37.896+0000 7f10e2b82b00 1 bucket sync caught up with source: 2021-09-17T10:11:37.880 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000001.84.5, , , , , , , , , ] 2021-09-17T10:11:37.880 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000001.84.5, , , , , , , , , ] 2021-09-17T10:11:37.880 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:37.896+0000 7f10e2b82b00 0 bucket checkpoint complete 2021-09-17T10:11:37.909 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-14 zones={test-zone3, test-zone2} 2021-09-17T10:11:37.917 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:37.967 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:37.968 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:37.974 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:38.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:38.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:38.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:38.299 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-14 zones={test-zone3, test-zone2} 2021-09-17T10:11:38.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:38.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:38.301 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:38.353 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:38.353 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:38.399 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:38.416+0000 7fdde290ab00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:38.399 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:38.416+0000 7fdde290ab00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:38.400 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:38.416+0000 7fdde290ab00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:11:38.400 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:38.416+0000 7fdde290ab00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:38.452 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:38.472+0000 7fdde290ab00 1 bucket sync caught up with source: 2021-09-17T10:11:38.452 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000001.126.5, , , , , , , , , ] 2021-09-17T10:11:38.453 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000001.126.5, , , , , , , , , ] 2021-09-17T10:11:38.453 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:38.472+0000 7fdde290ab00 0 bucket checkpoint complete 2021-09-17T10:11:38.478 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-15 zones={test-zone1, test-zone3} 2021-09-17T10:11:38.486 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:38.535 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:38.536 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:38.543 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:38.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:38.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:38.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:38.866 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-15 zones={test-zone1, test-zone3} 2021-09-17T10:11:38.866 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:38.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:38.868 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:38.918 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:38.970 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:38.990+0000 7f9a0eb0bb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:38.970 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:38.990+0000 7f9a0eb0bb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:38.971 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:38.990+0000 7f9a0eb0bb00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:11:38.971 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:38.990+0000 7f9a0eb0bb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:39.024 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:39.042+0000 7f9a0eb0bb00 1 bucket sync caught up with source: 2021-09-17T10:11:39.024 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000001.126.5, , , , , , , , , ] 2021-09-17T10:11:39.025 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000001.126.5, , , , , , , , , ] 2021-09-17T10:11:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:39.042+0000 7f9a0eb0bb00 0 bucket checkpoint complete 2021-09-17T10:11:39.054 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-15 zones={test-zone2, test-zone3} 2021-09-17T10:11:39.063 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:39.111 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:39.112 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:39.119 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-17T10:11:39.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:39.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:39.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-17T10:11:39.453 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-15 zones={test-zone2, test-zone3} 2021-09-17T10:11:39.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:39.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:39.468 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:39.518 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:39.566 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:39.586+0000 7f8fde8a8b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:39.567 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:39.586+0000 7f8fde8a8b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:39.567 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:39.586+0000 7f8fde8a8b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:11:39.567 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:39.586+0000 7f8fde8a8b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:39.621 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:39.642+0000 7f8fde8a8b00 1 waiting for incremental sync to catch up: 2021-09-17T10:11:39.621 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2021-09-17T10:11:39.621 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2021-09-17T10:11:44.623 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:44.642+0000 7f8fde8a8b00 1 bucket sync caught up with source: 2021-09-17T10:11:44.623 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2021-09-17T10:11:44.623 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2021-09-17T10:11:44.624 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:44.642+0000 7f8fde8a8b00 0 bucket checkpoint complete 2021-09-17T10:11:44.653 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-13 zones={test-zone2, test-zone1} 2021-09-17T10:11:44.663 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:44.711 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:44.773 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-13 zones={test-zone2, test-zone1} 2021-09-17T10:11:44.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:44.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:44.775 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:44.820 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:44.821 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:44.867 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:44.884+0000 7f1b794f9b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:44.868 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:44.884+0000 7f1b794f9b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:44.868 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:44.884+0000 7f1b794f9b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:11:44.868 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:44.884+0000 7f1b794f9b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:44.919 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:44.936+0000 7f1b794f9b00 1 bucket sync caught up with source: 2021-09-17T10:11:44.919 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2021-09-17T10:11:44.920 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2021-09-17T10:11:44.920 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:44.936+0000 7f1b794f9b00 0 bucket checkpoint complete 2021-09-17T10:11:44.944 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-13 zones={test-zone3, test-zone1} 2021-09-17T10:11:44.954 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:44.999 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:45.061 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-13 zones={test-zone3, test-zone1} 2021-09-17T10:11:45.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:45.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:45.063 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:45.113 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:45.114 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:45.161 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:45.182+0000 7f6ab3814b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:45.162 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:45.182+0000 7f6ab3814b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:45.162 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:45.182+0000 7f6ab3814b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:11:45.162 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:45.182+0000 7f6ab3814b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:45.218 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:45.238+0000 7f6ab3814b00 1 bucket sync caught up with source: 2021-09-17T10:11:45.218 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2021-09-17T10:11:45.218 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2021-09-17T10:11:45.219 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:45.238+0000 7f6ab3814b00 0 bucket checkpoint complete 2021-09-17T10:11:45.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:45.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:45.265 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:45.316 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:45.317 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:45.356 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.376+0000 7f26d6d68b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:45.356 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.376+0000 7f26d6d68b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:45.356 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.376+0000 7f26d6d68b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:11:45.357 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.376+0000 7f26d6d68b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:45.406 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.424+0000 7f26d6d68b00 1 bucket sync caught up with source: 2021-09-17T10:11:45.407 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000002.186.5, , , , , , , , , ] 2021-09-17T10:11:45.408 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000002.186.5, , , , , , , , , ] 2021-09-17T10:11:45.408 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.424+0000 7f26d6d68b00 0 bucket checkpoint complete 2021-09-17T10:11:45.435 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-14 zones={test-zone1, test-zone2} 2021-09-17T10:11:45.444 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:45.491 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:45.553 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-14 zones={test-zone1, test-zone2} 2021-09-17T10:11:45.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:45.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:45.555 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:45.605 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:45.605 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:45.655 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.672+0000 7ff0fa232b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:45.655 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.672+0000 7ff0fa232b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:45.656 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.672+0000 7ff0fa232b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:11:45.656 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.672+0000 7ff0fa232b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:45.710 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.728+0000 7ff0fa232b00 1 bucket sync caught up with source: 2021-09-17T10:11:45.710 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000002.186.5, , , , , , , , , ] 2021-09-17T10:11:45.710 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000002.186.5, , , , , , , , , ] 2021-09-17T10:11:45.711 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.728+0000 7ff0fa232b00 0 bucket checkpoint complete 2021-09-17T10:11:45.738 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-14 zones={test-zone3, test-zone2} 2021-09-17T10:11:45.748 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:45.799 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:45.861 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-14 zones={test-zone3, test-zone2} 2021-09-17T10:11:45.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:45.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:45.878 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:45.925 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:45.925 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:45.973 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.992+0000 7f4bb4e8cb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:45.974 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.992+0000 7f4bb4e8cb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:45.974 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.992+0000 7f4bb4e8cb00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:11:45.974 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:45.992+0000 7f4bb4e8cb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:46.021 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:46.040+0000 7f4bb4e8cb00 1 bucket sync caught up with source: 2021-09-17T10:11:46.021 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000002.192.5, , , , , , , , , ] 2021-09-17T10:11:46.021 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000002.192.5, , , , , , , , , ] 2021-09-17T10:11:46.022 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:46.040+0000 7f4bb4e8cb00 0 bucket checkpoint complete 2021-09-17T10:11:46.046 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-15 zones={test-zone1, test-zone3} 2021-09-17T10:11:46.054 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:46.099 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:46.157 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-15 zones={test-zone1, test-zone3} 2021-09-17T10:11:46.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:46.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:46.159 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:46.210 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:46.210 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:46.259 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:46.278+0000 7f87add5db00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:46.260 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:46.278+0000 7f87add5db00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:46.260 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:46.278+0000 7f87add5db00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:11:46.260 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:46.278+0000 7f87add5db00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:46.312 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:46.334+0000 7f87add5db00 1 bucket sync caught up with source: 2021-09-17T10:11:46.313 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000002.192.5, , , , , , , , , ] 2021-09-17T10:11:46.313 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000002.192.5, , , , , , , , , ] 2021-09-17T10:11:46.314 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:46.334+0000 7f87add5db00 0 bucket checkpoint complete 2021-09-17T10:11:46.341 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-15 zones={test-zone2, test-zone3} 2021-09-17T10:11:46.350 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:11:46.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:11:46.457 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-15 zones={test-zone2, test-zone3} 2021-09-17T10:11:46.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2021-09-17T10:11:46.458 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-16 2021-09-17T10:11:46.486 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-17 2021-09-17T10:11:46.534 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-18 2021-09-17T10:11:46.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:46.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:46.626 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:46.679 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:46.679 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:46.797 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:11:46.858 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:11:46.858 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:46.859 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:46.859 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:11:46.859 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:46.859 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:46.860 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873506.638251_114.1", 2021-09-17T10:11:46.860 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:46.638251Z" 2021-09-17T10:11:46.860 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:46.860 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:46.861 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:46.861 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:46.861 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:46.861 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:46.862 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:46.862 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:11:46.862 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:46.862 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:11:46.864 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:11:46.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:46.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:46.866 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:46.958 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:46.959 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:47.116 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:47.117 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:47.117 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:47.117 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:47.117 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:47.118 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:47.118 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:47.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:47.118 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:47.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:47.119 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:47.119 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:47.119 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:47.120 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:47.120 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:47.120 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:47.120 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:47.120 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:47.121 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:47.121 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:47.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:47.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:47.122 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:47.122 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:47.122 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:47.122 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873506.638251_114.1", 2021-09-17T10:11:47.122 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:47.123 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:47.123 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:47.123 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:46.638251Z", 2021-09-17T10:11:47.123 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:47.123 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:47.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:47.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:47.124 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:47.125 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:47.125 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:47.125 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:47.125 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:47.126 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:47.126 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:47.126 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:47.126 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:47.126 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:47.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:47.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:47.127 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:47.127 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:47.128 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:47.128 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:47.128 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:47.128 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:47.128 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:47.129 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:47.129 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:47.129 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:47.129 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:47.130 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:47.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:47.130 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:47.130 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:47.130 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:47.131 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:47.131 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:47.132 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:11:47.133 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:11:47.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:47.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:47.134 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:47.175 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:47.176 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:47.290 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:11:47.290 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:11:47.290 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:11:47.291 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:11:47.291 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:11:47.291 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:47.291 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:47.291 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:47.292 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:11:47.292 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:47.292 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:11:47.292 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:47.293 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:47.293 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:47.293 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:47.293 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:11:47.293 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:47.294 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:47.294 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:47.294 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:47.294 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:47.295 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:47.295 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:11:47.295 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:47.295 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:47.295 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873506.638251_114.1", 2021-09-17T10:11:47.296 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:47.296 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:47.296 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:47.296 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:46.638251Z", 2021-09-17T10:11:47.297 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:47.297 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:47.297 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:47.297 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:47.298 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:11:47.298 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:47.298 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:47.299 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:47.299 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:47.299 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:47.299 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:47.300 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:47.300 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:47.300 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:47.300 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:47.300 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:11:47.301 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:11:47.301 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:11:47.301 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:11:47.301 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:11:47.301 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:11:47.302 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:11:47.302 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:11:47.302 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:11:47.302 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:11:47.303 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:47.303 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:47.303 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:11:47.303 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:11:47.303 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:11:47.304 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:11:47.304 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:11:47.304 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:11:47.304 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:11:47.306 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:11:47.306 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:11:47.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:47.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:11:47.308 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:11:47.353 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:47.354 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:47.471 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:11:47.471 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:11:47.472 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:11:47.472 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:11:47.472 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:11:47.472 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:11:47.473 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:47.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:47.473 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:11:47.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:47.474 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:11:47.474 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:47.474 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:47.475 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:11:47.475 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:47.475 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:11:47.476 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:47.476 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:11:47.476 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:47.476 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:47.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:47.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:47.477 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:11:47.477 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:47.477 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:47.478 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873506.638251_114.1", 2021-09-17T10:11:47.478 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:47.478 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:47.478 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:47.478 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:46.638251Z", 2021-09-17T10:11:47.479 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:47.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:47.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:47.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:47.480 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:11:47.480 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:47.480 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:47.480 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:47.481 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:47.481 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:47.481 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:47.481 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:47.481 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:47.482 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:47.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:47.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:11:47.482 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:11:47.483 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:11:47.483 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:11:47.483 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:11:47.483 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:11:47.483 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:11:47.484 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:11:47.484 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:11:47.484 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:11:47.484 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:47.485 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:47.485 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:11:47.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:11:47.485 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:11:47.485 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:11:47.486 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:11:47.486 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:11:47.486 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:11:47.487 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:11:47.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:47.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:47.545 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:47.594 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:47.595 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:47.644 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:47.666+0000 7f44f64b8b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:47.645 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:47.666+0000 7f44f64b8b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:47.645 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:47.666+0000 7f44f64b8b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:11:47.645 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:47.666+0000 7f44f64b8b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:47.699 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:47.718+0000 7f44f64b8b00 1 waiting for incremental sync to catch up: 2021-09-17T10:11:47.699 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , , , ] 2021-09-17T10:11:47.700 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000002.7.10, , , , , ] 2021-09-17T10:11:52.702 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:52.722+0000 7f44f64b8b00 1 bucket sync caught up with source: 2021-09-17T10:11:52.702 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2021-09-17T10:11:52.703 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000002.7.10, , , , , ] 2021-09-17T10:11:52.703 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:52.722+0000 7f44f64b8b00 0 bucket checkpoint complete 2021-09-17T10:11:52.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:52.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:52.734 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:52.783 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:52.784 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:52.833 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:52.852+0000 7fc30bd32b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:52.834 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:52.852+0000 7fc30bd32b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:52.834 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:52.852+0000 7fc30bd32b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:11:52.834 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:52.852+0000 7fc30bd32b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:52.884 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:52.904+0000 7fc30bd32b00 1 bucket sync caught up with source: 2021-09-17T10:11:52.885 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2021-09-17T10:11:52.885 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2021-09-17T10:11:52.885 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:52.904+0000 7fc30bd32b00 0 bucket checkpoint complete 2021-09-17T10:11:52.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:52.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:52.911 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:52.963 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:52.963 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:53.009 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:53.030+0000 7f208e0c7b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:53.010 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:53.030+0000 7f208e0c7b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:53.010 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:53.030+0000 7f208e0c7b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:11:53.010 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:53.030+0000 7f208e0c7b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:53.066 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:53.086+0000 7f208e0c7b00 1 bucket sync caught up with source: 2021-09-17T10:11:53.067 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2021-09-17T10:11:53.067 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2021-09-17T10:11:53.067 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:53.086+0000 7f208e0c7b00 0 bucket checkpoint complete 2021-09-17T10:11:53.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:53.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:53.148 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:53.198 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:53.198 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:53.236 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:53.256+0000 7fd2b1db1b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:53.236 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:53.256+0000 7fd2b1db1b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:53.237 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:53.256+0000 7fd2b1db1b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:11:53.237 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:53.256+0000 7fd2b1db1b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:53.284 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:53.304+0000 7fd2b1db1b00 1 waiting for incremental sync to catch up: 2021-09-17T10:11:53.285 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000002.114.10, , , , , ] 2021-09-17T10:11:53.285 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000003.116.9, , , , , ] 2021-09-17T10:11:58.288 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.308+0000 7fd2b1db1b00 1 bucket sync caught up with source: 2021-09-17T10:11:58.288 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000006.122.9, , , , , ] 2021-09-17T10:11:58.289 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000003.116.9, , , , , ] 2021-09-17T10:11:58.289 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.308+0000 7fd2b1db1b00 0 bucket checkpoint complete 2021-09-17T10:11:58.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:58.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:58.312 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:58.406 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:58.407 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:58.453 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.472+0000 7fd5dbe04b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:58.453 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.472+0000 7fd5dbe04b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:58.453 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.472+0000 7fd5dbe04b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:11:58.454 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.472+0000 7fd5dbe04b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:58.507 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.524+0000 7fd5dbe04b00 1 bucket sync caught up with source: 2021-09-17T10:11:58.507 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000006.122.9, , , , , ] 2021-09-17T10:11:58.507 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000006.122.9, , , , , ] 2021-09-17T10:11:58.508 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.524+0000 7fd5dbe04b00 0 bucket checkpoint complete 2021-09-17T10:11:58.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:58.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:58.584 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:58.637 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:58.637 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:58.680 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.700+0000 7fae692a3b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:58.680 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.700+0000 7fae692a3b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:58.680 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.700+0000 7fae692a3b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:11:58.680 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.700+0000 7fae692a3b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:58.730 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.748+0000 7fae692a3b00 1 bucket sync caught up with source: 2021-09-17T10:11:58.730 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000002.145.10, , , , , ] 2021-09-17T10:11:58.730 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000002.145.10, , , , , ] 2021-09-17T10:11:58.731 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:11:58.748+0000 7fae692a3b00 0 bucket checkpoint complete 2021-09-17T10:11:58.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:58.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:11:58.754 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:11:58.807 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:11:58.808 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:11:58.855 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:58.874+0000 7ffa1b198b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:11:58.856 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:58.874+0000 7ffa1b198b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:11:58.856 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:58.874+0000 7ffa1b198b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:11:58.856 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:58.874+0000 7ffa1b198b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:11:58.902 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:11:58.922+0000 7ffa1b198b00 1 waiting for incremental sync to catch up: 2021-09-17T10:11:58.903 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , 00000000002.145.10, , , , , ] 2021-09-17T10:11:58.903 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000004.149.9, , , , , ] 2021-09-17T10:12:03.906 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:03.926+0000 7ffa1b198b00 1 bucket sync caught up with source: 2021-09-17T10:12:03.906 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , 00000000005.151.9, , , , , ] 2021-09-17T10:12:03.906 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000004.149.9, , , , , ] 2021-09-17T10:12:03.907 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:03.926+0000 7ffa1b198b00 0 bucket checkpoint complete 2021-09-17T10:12:05.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:05.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:05.595 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:05.651 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:05.651 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:05.687 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:05.706+0000 7f5f4d492b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:05.706+0000 7f5f4d492b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:05.706+0000 7f5f4d492b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:05.688 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:05.706+0000 7f5f4d492b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:05.740 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:05.762+0000 7f5f4d492b00 1 bucket sync caught up with source: 2021-09-17T10:12:05.741 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000010.24.5, , 00000000003.14.9, , 00000000011.26.5, , , , , ] 2021-09-17T10:12:05.741 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000010.24.5, , 00000000003.14.9, , 00000000011.26.5, , , , , ] 2021-09-17T10:12:05.741 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:05.762+0000 7f5f4d492b00 0 bucket checkpoint complete 2021-09-17T10:12:05.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:05.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:05.767 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:05.816 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:05.817 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:05.861 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:05.880+0000 7fd87e2e7b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:05.862 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:05.880+0000 7fd87e2e7b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:05.862 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:05.880+0000 7fd87e2e7b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:05.862 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:05.880+0000 7fd87e2e7b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:05.908 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:05.928+0000 7fd87e2e7b00 1 bucket sync caught up with source: 2021-09-17T10:12:05.908 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000010.24.5, , 00000000003.14.9, , 00000000011.26.5, , , , , ] 2021-09-17T10:12:05.909 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000010.24.5, , 00000000003.14.9, , 00000000011.26.5, , , , , ] 2021-09-17T10:12:05.909 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:05.928+0000 7fd87e2e7b00 0 bucket checkpoint complete 2021-09-17T10:12:05.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:05.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:05.936 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:05.987 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:05.987 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:06.034 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:06.054+0000 7fef007a7b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:06.035 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:06.054+0000 7fef007a7b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:06.035 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:06.054+0000 7fef007a7b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:12:06.035 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:06.054+0000 7fef007a7b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:06.091 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:06.110+0000 7fef007a7b00 1 bucket sync caught up with source: 2021-09-17T10:12:06.091 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000010.24.5, , 00000000003.14.9, , 00000000011.26.5, , , , , ] 2021-09-17T10:12:06.092 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000010.24.5, , 00000000003.14.9, , 00000000011.26.5, , , , , ] 2021-09-17T10:12:06.092 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:06.110+0000 7fef007a7b00 0 bucket checkpoint complete 2021-09-17T10:12:06.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:06.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:06.122 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:06.174 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:06.174 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:06.223 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.240+0000 7f3e4ccfcb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:06.223 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.240+0000 7f3e4ccfcb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:06.224 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.240+0000 7f3e4ccfcb00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:06.224 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.240+0000 7f3e4ccfcb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:06.281 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.300+0000 7f3e4ccfcb00 1 bucket sync caught up with source: 2021-09-17T10:12:06.281 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000003.99.9, , 00000000010.174.5, , 00000000007.131.9, , , , , ] 2021-09-17T10:12:06.281 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000003.99.9, , 00000000010.174.5, , 00000000007.131.9, , , , , ] 2021-09-17T10:12:06.281 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.300+0000 7f3e4ccfcb00 0 bucket checkpoint complete 2021-09-17T10:12:06.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:06.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:06.308 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:06.352 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:06.352 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:06.403 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.420+0000 7fe285e39b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:06.403 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.420+0000 7fe285e39b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:06.403 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.420+0000 7fe285e39b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:06.404 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.420+0000 7fe285e39b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:06.458 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.476+0000 7fe285e39b00 1 bucket sync caught up with source: 2021-09-17T10:12:06.458 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000003.99.9, , 00000000010.174.5, , 00000000007.131.9, , , , , ] 2021-09-17T10:12:06.459 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000003.99.9, , 00000000010.174.5, , 00000000007.131.9, , , , , ] 2021-09-17T10:12:06.459 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.476+0000 7fe285e39b00 0 bucket checkpoint complete 2021-09-17T10:12:06.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:06.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:06.489 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:06.538 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:06.538 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:06.584 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.604+0000 7fa9fb071b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:06.584 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.604+0000 7fa9fb071b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:06.585 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.604+0000 7fa9fb071b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:06.585 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.604+0000 7fa9fb071b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:06.639 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.660+0000 7fa9fb071b00 1 bucket sync caught up with source: 2021-09-17T10:12:06.640 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000003.95.9, , 00000000003.200.9, , 00000000014.137.5, , , , , ] 2021-09-17T10:12:06.640 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000003.95.9, , 00000000003.200.9, , 00000000014.137.5, , , , , ] 2021-09-17T10:12:06.640 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:06.660+0000 7fa9fb071b00 0 bucket checkpoint complete 2021-09-17T10:12:06.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:06.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:06.666 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:06.714 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:06.715 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:06.765 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:06.786+0000 7fa832e59b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:06.765 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:06.786+0000 7fa832e59b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:06.766 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:06.786+0000 7fa832e59b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:06.766 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:06.786+0000 7fa832e59b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:06.822 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:06.842+0000 7fa832e59b00 1 bucket sync caught up with source: 2021-09-17T10:12:06.822 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000003.95.9, , 00000000003.200.9, , 00000000014.137.5, , , , , ] 2021-09-17T10:12:06.823 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000003.95.9, , 00000000003.200.9, , 00000000014.137.5, , , , , ] 2021-09-17T10:12:06.823 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:06.842+0000 7fa832e59b00 0 bucket checkpoint complete 2021-09-17T10:12:06.852 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-16 zones={test-zone2, test-zone1} 2021-09-17T10:12:06.862 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:06.911 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:06.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:06.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:06.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:06.913 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:06.921 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:06.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:06.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:06.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:06.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:06.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:07.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:07.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:07.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:07.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:07.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:07.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:07.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:07.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:07.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:07.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:08.006 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-16 zones={test-zone2, test-zone1} 2021-09-17T10:12:08.007 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-16 zones={test-zone3, test-zone1} 2021-09-17T10:12:08.015 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:08.063 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:08.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:08.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:08.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:08.065 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:08.074 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:08.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:08.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:08.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:08.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:08.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:08.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:08.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:08.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:08.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:08.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:08.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:08.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:08.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:08.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:08.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:09.107 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-16 zones={test-zone3, test-zone1} 2021-09-17T10:12:09.108 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-16 zones={test-zone3, test-zone2} 2021-09-17T10:12:09.116 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:09.121 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:09.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:09.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:09.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:09.123 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:09.171 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:09.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:09.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:09.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:09.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:09.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:09.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:09.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:09.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:09.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:09.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:09.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:09.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:09.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:09.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:10.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:10.209 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-16 zones={test-zone3, test-zone2} 2021-09-17T10:12:10.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:10.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:10.211 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:10.263 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:10.263 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:10.306 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:10.326+0000 7f79c5045b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:10.307 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:10.326+0000 7f79c5045b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:10.307 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:10.326+0000 7f79c5045b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:10.307 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:10.326+0000 7f79c5045b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:10.356 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:10.378+0000 7f79c5045b00 1 bucket sync caught up with source: 2021-09-17T10:12:10.357 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000010.26.5, , 00000000006.15.5, , 00000000007.16.9, , , , , ] 2021-09-17T10:12:10.357 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000010.26.5, , 00000000006.15.5, , 00000000007.16.9, , , , , ] 2021-09-17T10:12:10.357 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:10.378+0000 7f79c5045b00 0 bucket checkpoint complete 2021-09-17T10:12:10.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:10.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:10.385 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:10.436 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:10.436 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:10.483 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:10.504+0000 7f0ad1bc0b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:10.484 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:10.504+0000 7f0ad1bc0b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:10.484 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:10.504+0000 7f0ad1bc0b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:10.484 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:10.504+0000 7f0ad1bc0b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:10.534 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:10.552+0000 7f0ad1bc0b00 1 bucket sync caught up with source: 2021-09-17T10:12:10.534 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000010.26.5, , 00000000006.15.5, , 00000000007.16.9, , , , , ] 2021-09-17T10:12:10.535 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000010.26.5, , 00000000006.15.5, , 00000000007.16.9, , , , , ] 2021-09-17T10:12:10.535 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:10.552+0000 7f0ad1bc0b00 0 bucket checkpoint complete 2021-09-17T10:12:10.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:10.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:10.562 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:10.612 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:10.612 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:10.659 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:10.678+0000 7f4e15b49b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:10.660 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:10.678+0000 7f4e15b49b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:10.660 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:10.678+0000 7f4e15b49b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:12:10.660 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:10.678+0000 7f4e15b49b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:10.716 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:10.738+0000 7f4e15b49b00 1 bucket sync caught up with source: 2021-09-17T10:12:10.716 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000010.26.5, , 00000000006.15.5, , 00000000007.16.9, , , , , ] 2021-09-17T10:12:10.716 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000010.26.5, , 00000000006.15.5, , 00000000007.16.9, , , , , ] 2021-09-17T10:12:10.717 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:10.738+0000 7f4e15b49b00 0 bucket checkpoint complete 2021-09-17T10:12:10.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:10.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:10.747 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:10.797 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:10.798 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:10.842 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:10.860+0000 7f6d7bf80b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:10.843 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:10.860+0000 7f6d7bf80b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:10.843 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:10.860+0000 7f6d7bf80b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:10.843 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:10.860+0000 7f6d7bf80b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:10.899 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:10.916+0000 7f6d7bf80b00 1 bucket sync caught up with source: 2021-09-17T10:12:10.899 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000005.185.5, , 00000000010.101.5, , 00000000009.128.9, , , , , ] 2021-09-17T10:12:10.899 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000005.185.5, , 00000000010.101.5, , 00000000009.128.9, , , , , ] 2021-09-17T10:12:10.900 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:10.916+0000 7f6d7bf80b00 0 bucket checkpoint complete 2021-09-17T10:12:10.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:10.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:10.931 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:10.976 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:10.976 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:11.023 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:11.040+0000 7faa5db84b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:11.023 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:11.040+0000 7faa5db84b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:11.024 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:11.040+0000 7faa5db84b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:11.024 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:11.040+0000 7faa5db84b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:11.077 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:11.096+0000 7faa5db84b00 1 bucket sync caught up with source: 2021-09-17T10:12:11.078 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000005.185.5, , 00000000010.101.5, , 00000000009.128.9, , , , , ] 2021-09-17T10:12:11.078 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000005.185.5, , 00000000010.101.5, , 00000000009.128.9, , , , , ] 2021-09-17T10:12:11.078 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:11.096+0000 7faa5db84b00 0 bucket checkpoint complete 2021-09-17T10:12:11.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:11.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:11.108 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:11.158 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:11.158 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:11.211 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:11.228+0000 7f76cc085b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:11.211 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:11.228+0000 7f76cc085b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:11.212 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:11.228+0000 7f76cc085b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:11.212 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:11.228+0000 7f76cc085b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:11.265 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:11.284+0000 7f76cc085b00 1 bucket sync caught up with source: 2021-09-17T10:12:11.265 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000005.164.5, , 00000000009.91.5, , 00000000014.127.5, , , , , ] 2021-09-17T10:12:11.266 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000005.164.5, , 00000000009.91.5, , 00000000014.127.5, , , , , ] 2021-09-17T10:12:11.266 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:11.284+0000 7f76cc085b00 0 bucket checkpoint complete 2021-09-17T10:12:11.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:11.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:11.292 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:11.343 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:11.344 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:11.406+0000 7f0dcd837b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:11.406+0000 7f0dcd837b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:11.406+0000 7f0dcd837b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:11.406+0000 7f0dcd837b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:11.442 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:11.462+0000 7f0dcd837b00 1 bucket sync caught up with source: 2021-09-17T10:12:11.443 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000005.164.5, , 00000000009.91.5, , 00000000014.127.5, , , , , ] 2021-09-17T10:12:11.443 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000005.164.5, , 00000000009.91.5, , 00000000014.127.5, , , , , ] 2021-09-17T10:12:11.443 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:11.462+0000 7f0dcd837b00 0 bucket checkpoint complete 2021-09-17T10:12:11.472 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-17 zones={test-zone2, test-zone1} 2021-09-17T10:12:11.482 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:11.535 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:11.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:11.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:11.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:11.537 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:11.544 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:11.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:11.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:11.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:11.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:11.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:11.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:11.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:11.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:11.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:12.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:12.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:12.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:12.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:12.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:12.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:12.582 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-17 zones={test-zone2, test-zone1} 2021-09-17T10:12:12.583 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-17 zones={test-zone3, test-zone1} 2021-09-17T10:12:12.591 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:12.639 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:12.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:12.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:12.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:12.642 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:12.649 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:12.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:12.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:12.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:12.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:12.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:12.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:12.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:12.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:13.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:13.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:13.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:13.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:13.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:13.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:13.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:13.670 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-17 zones={test-zone3, test-zone1} 2021-09-17T10:12:13.671 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-17 zones={test-zone3, test-zone2} 2021-09-17T10:12:13.680 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:13.731 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:13.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:13.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:13.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:13.733 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:13.740 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:13.741 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:13.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:13.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:13.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:13.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:13.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:13.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:13.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:14.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:14.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:14.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:14.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:14.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:14.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:14.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:14.785 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-17 zones={test-zone3, test-zone2} 2021-09-17T10:12:14.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:14.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:14.787 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:14.838 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:14.838 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:14.881 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:14.902+0000 7ff26a1d0b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:14.881 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:14.902+0000 7ff26a1d0b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:14.882 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:14.902+0000 7ff26a1d0b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:14.882 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:14.902+0000 7ff26a1d0b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:14.932 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:14.954+0000 7ff26a1d0b00 1 bucket sync caught up with source: 2021-09-17T10:12:14.932 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000010.29.5, , 00000000003.19.9, , 00000000007.18.9, , , , , ] 2021-09-17T10:12:14.932 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000010.29.5, , 00000000003.19.9, , 00000000007.18.9, , , , , ] 2021-09-17T10:12:14.933 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:14.954+0000 7ff26a1d0b00 0 bucket checkpoint complete 2021-09-17T10:12:14.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:14.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:14.961 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:15.009 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:15.009 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:15.059 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.076+0000 7fa6148d6b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:15.059 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.076+0000 7fa6148d6b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:15.060 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.076+0000 7fa6148d6b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:15.060 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.076+0000 7fa6148d6b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:15.111 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.128+0000 7fa6148d6b00 1 bucket sync caught up with source: 2021-09-17T10:12:15.111 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000010.29.5, , 00000000003.19.9, , 00000000007.18.9, , , , , ] 2021-09-17T10:12:15.112 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000010.29.5, , 00000000003.19.9, , 00000000007.18.9, , , , , ] 2021-09-17T10:12:15.112 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.128+0000 7fa6148d6b00 0 bucket checkpoint complete 2021-09-17T10:12:15.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:15.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:15.135 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:15.184 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:15.185 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:15.232 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:15.254+0000 7f426c9a5b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:15.233 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:15.254+0000 7f426c9a5b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:15.233 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:15.254+0000 7f426c9a5b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:12:15.233 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:15.254+0000 7f426c9a5b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:15.287 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:15.310+0000 7f426c9a5b00 1 bucket sync caught up with source: 2021-09-17T10:12:15.288 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000010.29.5, , 00000000003.19.9, , 00000000007.18.9, , , , , ] 2021-09-17T10:12:15.288 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000010.29.5, , 00000000003.19.9, , 00000000007.18.9, , , , , ] 2021-09-17T10:12:15.288 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:15.310+0000 7f426c9a5b00 0 bucket checkpoint complete 2021-09-17T10:12:15.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:15.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:15.317 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:15.366 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:15.366 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:15.415 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.432+0000 7f1212cdeb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:15.415 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.432+0000 7f1212cdeb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:15.416 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.432+0000 7f1212cdeb00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:15.416 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.432+0000 7f1212cdeb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:15.469 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.488+0000 7f1212cdeb00 1 bucket sync caught up with source: 2021-09-17T10:12:15.469 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000003.237.9, , 00000000010.314.5, , 00000000007.74.9, , , , , ] 2021-09-17T10:12:15.470 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000003.237.9, , 00000000010.314.5, , 00000000007.74.9, , , , , ] 2021-09-17T10:12:15.470 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.488+0000 7f1212cdeb00 0 bucket checkpoint complete 2021-09-17T10:12:15.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:15.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:15.501 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:15.550 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:15.551 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:15.599 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.620+0000 7ff2f45dbb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:15.600 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.620+0000 7ff2f45dbb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:15.600 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.620+0000 7ff2f45dbb00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:15.600 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.620+0000 7ff2f45dbb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:15.654 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.672+0000 7ff2f45dbb00 1 bucket sync caught up with source: 2021-09-17T10:12:15.654 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000003.237.9, , 00000000010.314.5, , 00000000007.74.9, , , , , ] 2021-09-17T10:12:15.654 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000003.237.9, , 00000000010.314.5, , 00000000007.74.9, , , , , ] 2021-09-17T10:12:15.655 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.672+0000 7ff2f45dbb00 0 bucket checkpoint complete 2021-09-17T10:12:15.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:15.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:15.683 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:15.729 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:15.730 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:15.777 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.796+0000 7f1378900b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:15.777 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.796+0000 7f1378900b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:15.777 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.796+0000 7f1378900b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:15.778 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.796+0000 7f1378900b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:15.831 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.848+0000 7f1378900b00 1 bucket sync caught up with source: 2021-09-17T10:12:15.831 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000003.218.9, , 00000000003.294.9, , 00000000015.171.5, , , , , ] 2021-09-17T10:12:15.832 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000003.218.9, , 00000000003.294.9, , 00000000015.171.5, , , , , ] 2021-09-17T10:12:15.832 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:15.848+0000 7f1378900b00 0 bucket checkpoint complete 2021-09-17T10:12:15.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:15.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:15.858 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:15.908 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:15.909 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:15.961 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:15.982+0000 7f6f8789cb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:15.961 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:15.982+0000 7f6f8789cb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:15.962 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:15.982+0000 7f6f8789cb00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:15.962 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:15.982+0000 7f6f8789cb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:16.016 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:16.038+0000 7f6f8789cb00 1 bucket sync caught up with source: 2021-09-17T10:12:16.016 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000003.218.9, , 00000000003.294.9, , 00000000015.171.5, , , , , ] 2021-09-17T10:12:16.017 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000003.218.9, , 00000000003.294.9, , 00000000015.171.5, , , , , ] 2021-09-17T10:12:16.017 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:16.038+0000 7f6f8789cb00 0 bucket checkpoint complete 2021-09-17T10:12:16.046 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-18 zones={test-zone2, test-zone1} 2021-09-17T10:12:16.055 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:16.107 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:16.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:16.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:16.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:16.109 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:16.116 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:16.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:16.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:16.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:16.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:16.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:16.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:16.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:16.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:16.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:16.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:16.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:16.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:16.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:16.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:17.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:17.166 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-18 zones={test-zone2, test-zone1} 2021-09-17T10:12:17.166 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-18 zones={test-zone3, test-zone1} 2021-09-17T10:12:17.174 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:17.179 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:17.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:17.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:17.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:17.180 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:17.185 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:17.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:17.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:17.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:17.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:17.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:17.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:17.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:17.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:17.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:17.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:17.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:17.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:17.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:17.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:18.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:18.190 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-18 zones={test-zone3, test-zone1} 2021-09-17T10:12:18.190 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-18 zones={test-zone3, test-zone2} 2021-09-17T10:12:18.199 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:18.247 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:18.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:18.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:18.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:18.248 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:18.256 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:18.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:18.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:18.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:18.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:18.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:18.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:18.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:18.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:18.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:18.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:18.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:18.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:19.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:19.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:19.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:19.302 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-18 zones={test-zone3, test-zone2} 2021-09-17T10:12:19.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:19.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:19.899 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:19.954 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:19.954 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:20.014+0000 7f08e039cb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:20.014+0000 7f08e039cb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:20.014+0000 7f08e039cb00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:20.014+0000 7f08e039cb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:20.043 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:20.066+0000 7f08e039cb00 1 bucket sync caught up with source: 2021-09-17T10:12:20.044 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000011.30.5, , 00000000004.17.5, , 00000000012.29.5, , , , , ] 2021-09-17T10:12:20.044 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000011.30.5, , 00000000004.17.5, , 00000000012.29.5, , , , , ] 2021-09-17T10:12:20.044 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:20.066+0000 7f08e039cb00 0 bucket checkpoint complete 2021-09-17T10:12:20.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:20.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:20.072 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:20.121 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:20.122 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:20.162 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.180+0000 7f9c662a1b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:20.163 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.180+0000 7f9c662a1b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:20.163 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.180+0000 7f9c662a1b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:20.163 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.180+0000 7f9c662a1b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:20.209 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.228+0000 7f9c662a1b00 1 bucket sync caught up with source: 2021-09-17T10:12:20.210 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000011.30.5, , 00000000004.17.5, , 00000000012.29.5, , , , , ] 2021-09-17T10:12:20.210 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000011.30.5, , 00000000004.17.5, , 00000000012.29.5, , , , , ] 2021-09-17T10:12:20.210 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.228+0000 7f9c662a1b00 0 bucket checkpoint complete 2021-09-17T10:12:20.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:20.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:20.234 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:20.284 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:20.284 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:20.329 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:20.350+0000 7fded8ff2b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:20.330 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:20.350+0000 7fded8ff2b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:20.330 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:20.350+0000 7fded8ff2b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:12:20.330 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:20.350+0000 7fded8ff2b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:20.382 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:20.402+0000 7fded8ff2b00 1 bucket sync caught up with source: 2021-09-17T10:12:20.383 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000011.30.5, , 00000000004.17.5, , 00000000012.29.5, , , , , ] 2021-09-17T10:12:20.383 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000011.30.5, , 00000000004.17.5, , 00000000012.29.5, , , , , ] 2021-09-17T10:12:20.383 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:20.402+0000 7fded8ff2b00 0 bucket checkpoint complete 2021-09-17T10:12:20.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:20.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:20.412 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:20.461 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:20.461 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:20.509 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.528+0000 7fb281495b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:20.509 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.528+0000 7fb281495b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:20.509 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.528+0000 7fb281495b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:20.510 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.528+0000 7fb281495b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:20.563 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.580+0000 7fb281495b00 1 bucket sync caught up with source: 2021-09-17T10:12:20.563 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000004.102.5, , 00000000011.180.5, , 00000000008.134.5, , , , , ] 2021-09-17T10:12:20.563 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000004.102.5, , 00000000011.180.5, , 00000000008.134.5, , , , , ] 2021-09-17T10:12:20.564 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.580+0000 7fb281495b00 0 bucket checkpoint complete 2021-09-17T10:12:20.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:20.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:20.594 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:20.643 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:20.644 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:20.692 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.712+0000 7f7ac9d1bb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:20.692 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.712+0000 7f7ac9d1bb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:20.693 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.712+0000 7f7ac9d1bb00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:20.693 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.712+0000 7f7ac9d1bb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:20.748 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.768+0000 7f7ac9d1bb00 1 bucket sync caught up with source: 2021-09-17T10:12:20.748 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000004.102.5, , 00000000011.180.5, , 00000000008.134.5, , , , , ] 2021-09-17T10:12:20.749 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000004.102.5, , 00000000011.180.5, , 00000000008.134.5, , , , , ] 2021-09-17T10:12:20.749 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.768+0000 7f7ac9d1bb00 0 bucket checkpoint complete 2021-09-17T10:12:20.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:20.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:20.779 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:20.825 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:20.826 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:20.870 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.888+0000 7f549564eb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:20.871 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.888+0000 7f549564eb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:20.871 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.888+0000 7f549564eb00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:20.871 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.888+0000 7f549564eb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:20.925 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.944+0000 7f549564eb00 1 bucket sync caught up with source: 2021-09-17T10:12:20.925 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000004.98.5, , 00000000004.203.5, , 00000000015.140.5, , , , , ] 2021-09-17T10:12:20.925 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000004.98.5, , 00000000004.203.5, , 00000000015.140.5, , , , , ] 2021-09-17T10:12:20.926 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:20.944+0000 7f549564eb00 0 bucket checkpoint complete 2021-09-17T10:12:20.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:20.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:20.954 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:21.004 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:21.004 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:21.057 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:21.078+0000 7f8d38556b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:21.058 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:21.078+0000 7f8d38556b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:21.058 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:21.078+0000 7f8d38556b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:21.058 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:21.078+0000 7f8d38556b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:21.111 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:21.134+0000 7f8d38556b00 1 bucket sync caught up with source: 2021-09-17T10:12:21.112 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000004.98.5, , 00000000004.203.5, , 00000000015.140.5, , , , , ] 2021-09-17T10:12:21.112 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000004.98.5, , 00000000004.203.5, , 00000000015.140.5, , , , , ] 2021-09-17T10:12:21.112 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:21.134+0000 7f8d38556b00 0 bucket checkpoint complete 2021-09-17T10:12:21.141 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-16 zones={test-zone2, test-zone1} 2021-09-17T10:12:21.151 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:21.199 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:21.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:21.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:21.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:21.201 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:21.208 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:21.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:21.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:21.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:21.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:21.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:21.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:21.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:21.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:21.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:21.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:21.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:21.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:21.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:22.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:22.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:22.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-16 zones={test-zone2, test-zone1} 2021-09-17T10:12:22.243 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-16 zones={test-zone3, test-zone1} 2021-09-17T10:12:22.251 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:22.299 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:22.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:22.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:22.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:22.301 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:22.308 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:22.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:22.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:22.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:22.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:22.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:22.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:22.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:22.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:22.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:22.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:22.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:22.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:23.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:23.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:23.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:23.355 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-16 zones={test-zone3, test-zone1} 2021-09-17T10:12:23.355 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-16 zones={test-zone3, test-zone2} 2021-09-17T10:12:23.365 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:23.411 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:23.411 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:23.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:23.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:23.413 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:23.421 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:23.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:23.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:23.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:23.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:23.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:23.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:23.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:23.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:23.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:23.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:23.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:24.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:24.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:24.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:24.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:24.449 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-16 zones={test-zone3, test-zone2} 2021-09-17T10:12:24.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:24.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:24.451 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:24.500 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:24.501 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:24.544 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:24.566+0000 7f1cb96f0b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:24.545 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:24.566+0000 7f1cb96f0b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:24.545 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:24.566+0000 7f1cb96f0b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:24.545 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:24.566+0000 7f1cb96f0b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:24.600 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:24.622+0000 7f1cb96f0b00 1 bucket sync caught up with source: 2021-09-17T10:12:24.600 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000011.31.5, , 00000000007.18.5, , 00000000008.19.5, , , , , ] 2021-09-17T10:12:24.600 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000011.31.5, , 00000000007.18.5, , 00000000008.19.5, , , , , ] 2021-09-17T10:12:24.601 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:24.622+0000 7f1cb96f0b00 0 bucket checkpoint complete 2021-09-17T10:12:24.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:24.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:24.631 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:24.682 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:24.682 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:24.731 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:24.748+0000 7fd3e4c80b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:24.731 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:24.748+0000 7fd3e4c80b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:24.731 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:24.748+0000 7fd3e4c80b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:24.732 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:24.748+0000 7fd3e4c80b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:24.783 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:24.804+0000 7fd3e4c80b00 1 bucket sync caught up with source: 2021-09-17T10:12:24.784 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000011.31.5, , 00000000007.18.5, , 00000000008.19.5, , , , , ] 2021-09-17T10:12:24.784 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000011.31.5, , 00000000007.18.5, , 00000000008.19.5, , , , , ] 2021-09-17T10:12:24.784 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:24.804+0000 7fd3e4c80b00 0 bucket checkpoint complete 2021-09-17T10:12:24.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:24.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:24.809 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:24.858 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:24.859 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:24.906 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:24.926+0000 7f9d199d2b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:24.907 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:24.926+0000 7f9d199d2b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:24.907 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:24.926+0000 7f9d199d2b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:12:24.907 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:24.926+0000 7f9d199d2b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:24.961 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:24.982+0000 7f9d199d2b00 1 bucket sync caught up with source: 2021-09-17T10:12:24.961 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000011.31.5, , 00000000007.18.5, , 00000000008.19.5, , , , , ] 2021-09-17T10:12:24.961 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000011.31.5, , 00000000007.18.5, , 00000000008.19.5, , , , , ] 2021-09-17T10:12:24.961 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:24.982+0000 7f9d199d2b00 0 bucket checkpoint complete 2021-09-17T10:12:24.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:24.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:24.992 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:25.043 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:25.044 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:25.090 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.108+0000 7f31d4a99b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:25.091 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.108+0000 7f31d4a99b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:25.091 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.108+0000 7f31d4a99b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:25.091 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.108+0000 7f31d4a99b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:25.141 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.160+0000 7f31d4a99b00 1 bucket sync caught up with source: 2021-09-17T10:12:25.142 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000006.189.5, , 00000000011.106.5, , 00000000010.131.5, , , , , ] 2021-09-17T10:12:25.142 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000006.189.5, , 00000000011.106.5, , 00000000010.131.5, , , , , ] 2021-09-17T10:12:25.142 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.160+0000 7f31d4a99b00 0 bucket checkpoint complete 2021-09-17T10:12:25.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:25.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:25.172 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:25.221 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:25.221 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:25.264 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.284+0000 7f82087a5b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:25.264 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.284+0000 7f82087a5b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:25.264 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.284+0000 7f82087a5b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:25.265 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.284+0000 7f82087a5b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:25.315 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.336+0000 7f82087a5b00 1 bucket sync caught up with source: 2021-09-17T10:12:25.315 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000006.189.5, , 00000000011.106.5, , 00000000010.131.5, , , , , ] 2021-09-17T10:12:25.316 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000006.189.5, , 00000000011.106.5, , 00000000010.131.5, , , , , ] 2021-09-17T10:12:25.316 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.336+0000 7f82087a5b00 0 bucket checkpoint complete 2021-09-17T10:12:25.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:25.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:25.346 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:25.394 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:25.395 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:25.440 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.460+0000 7f02b3b5cb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:25.441 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.460+0000 7f02b3b5cb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:25.441 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.460+0000 7f02b3b5cb00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:25.441 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.460+0000 7f02b3b5cb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:25.492 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.512+0000 7f02b3b5cb00 1 bucket sync caught up with source: 2021-09-17T10:12:25.492 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000006.168.5, , 00000000010.94.5, , 00000000015.130.5, , , , , ] 2021-09-17T10:12:25.492 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000006.168.5, , 00000000010.94.5, , 00000000015.130.5, , , , , ] 2021-09-17T10:12:25.493 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:25.512+0000 7f02b3b5cb00 0 bucket checkpoint complete 2021-09-17T10:12:25.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:25.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:25.521 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:25.570 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:25.570 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:25.619 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:25.638+0000 7fe251df9b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:25.619 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:25.638+0000 7fe251df9b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:25.619 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:25.638+0000 7fe251df9b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:25.620 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:25.638+0000 7fe251df9b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:25.676 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:25.698+0000 7fe251df9b00 1 bucket sync caught up with source: 2021-09-17T10:12:25.676 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000006.168.5, , 00000000010.94.5, , 00000000015.130.5, , , , , ] 2021-09-17T10:12:25.676 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000006.168.5, , 00000000010.94.5, , 00000000015.130.5, , , , , ] 2021-09-17T10:12:25.676 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:25.698+0000 7fe251df9b00 0 bucket checkpoint complete 2021-09-17T10:12:25.705 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-17 zones={test-zone2, test-zone1} 2021-09-17T10:12:25.714 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:25.763 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:25.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:25.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:25.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:25.765 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:25.772 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:25.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:25.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:25.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:25.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:25.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:25.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:25.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:26.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:26.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:26.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:26.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:26.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:26.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:26.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:26.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:26.814 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-17 zones={test-zone2, test-zone1} 2021-09-17T10:12:26.814 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-17 zones={test-zone3, test-zone1} 2021-09-17T10:12:26.822 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:26.827 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:26.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:26.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:26.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:26.829 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:26.883 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:26.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:26.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:26.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:26.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:26.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:27.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:27.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:27.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:27.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:27.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:27.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:27.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:27.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:27.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:27.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:27.897 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-17 zones={test-zone3, test-zone1} 2021-09-17T10:12:27.898 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-17 zones={test-zone3, test-zone2} 2021-09-17T10:12:27.907 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:27.955 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:27.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:27.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:27.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:27.957 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:27.965 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:27.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:27.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:27.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:28.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:28.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:28.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:28.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:28.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:28.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:28.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:28.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:28.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:28.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:28.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:28.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:28.978 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-17 zones={test-zone3, test-zone2} 2021-09-17T10:12:28.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:28.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:28.982 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:29.030 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:29.030 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:29.084 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:29.106+0000 7f10b6149b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:29.084 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:29.106+0000 7f10b6149b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:29.085 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:29.106+0000 7f10b6149b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:29.085 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:29.106+0000 7f10b6149b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:29.140 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:29.162+0000 7f10b6149b00 1 bucket sync caught up with source: 2021-09-17T10:12:29.141 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000011.34.5, , 00000000004.22.5, , 00000000008.21.5, , , , , ] 2021-09-17T10:12:29.141 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000011.34.5, , 00000000004.22.5, , 00000000008.21.5, , , , , ] 2021-09-17T10:12:29.141 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:29.162+0000 7f10b6149b00 0 bucket checkpoint complete 2021-09-17T10:12:29.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:29.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:29.172 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:29.221 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:29.222 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:29.270 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.288+0000 7f995a200b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:29.271 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.288+0000 7f995a200b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:29.271 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.288+0000 7f995a200b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:29.271 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.288+0000 7f995a200b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:29.323 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.344+0000 7f995a200b00 1 bucket sync caught up with source: 2021-09-17T10:12:29.323 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000011.34.5, , 00000000004.22.5, , 00000000008.21.5, , , , , ] 2021-09-17T10:12:29.324 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000011.34.5, , 00000000004.22.5, , 00000000008.21.5, , , , , ] 2021-09-17T10:12:29.324 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.344+0000 7f995a200b00 0 bucket checkpoint complete 2021-09-17T10:12:29.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:29.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:29.349 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:29.399 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:29.400 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:29.448 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:29.470+0000 7f018e0ccb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:29.448 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:29.470+0000 7f018e0ccb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:29.448 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:29.470+0000 7f018e0ccb00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:12:29.449 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:29.470+0000 7f018e0ccb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:29.502 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:29.522+0000 7f018e0ccb00 1 bucket sync caught up with source: 2021-09-17T10:12:29.502 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000011.34.5, , 00000000004.22.5, , 00000000008.21.5, , , , , ] 2021-09-17T10:12:29.503 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000011.34.5, , 00000000004.22.5, , 00000000008.21.5, , , , , ] 2021-09-17T10:12:29.503 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:29.522+0000 7f018e0ccb00 0 bucket checkpoint complete 2021-09-17T10:12:29.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:29.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:29.534 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:29.584 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:29.584 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:29.631 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.652+0000 7fa92ee9fb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:29.631 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.652+0000 7fa92ee9fb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:29.632 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.652+0000 7fa92ee9fb00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:29.632 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.652+0000 7fa92ee9fb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:29.689 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.708+0000 7fa92ee9fb00 1 bucket sync caught up with source: 2021-09-17T10:12:29.690 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000004.240.5, , 00000000011.320.5, , 00000000008.77.5, , , , , ] 2021-09-17T10:12:29.690 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000004.240.5, , 00000000011.320.5, , 00000000008.77.5, , , , , ] 2021-09-17T10:12:29.690 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.708+0000 7fa92ee9fb00 0 bucket checkpoint complete 2021-09-17T10:12:29.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:29.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:29.721 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:29.769 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:29.769 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:29.819 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.840+0000 7f369c96db00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:29.820 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.840+0000 7f369c96db00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:29.820 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.840+0000 7f369c96db00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:29.820 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.840+0000 7f369c96db00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:29.875 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.892+0000 7f369c96db00 1 bucket sync caught up with source: 2021-09-17T10:12:29.875 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000004.240.5, , 00000000011.320.5, , 00000000008.77.5, , , , , ] 2021-09-17T10:12:29.875 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000004.240.5, , 00000000011.320.5, , 00000000008.77.5, , , , , ] 2021-09-17T10:12:29.876 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:29.892+0000 7f369c96db00 0 bucket checkpoint complete 2021-09-17T10:12:29.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:29.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:29.905 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:29.951 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:29.952 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:29.999 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:30.020+0000 7f8b5457eb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:29.999 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:30.020+0000 7f8b5457eb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:30.000 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:30.020+0000 7f8b5457eb00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:30.000 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:30.020+0000 7f8b5457eb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:30.053 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:30.072+0000 7f8b5457eb00 1 bucket sync caught up with source: 2021-09-17T10:12:30.054 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, 00000000004.221.5, , 00000000004.297.5, , 00000000016.174.5, , , , , ] 2021-09-17T10:12:30.054 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, 00000000004.221.5, , 00000000004.297.5, , 00000000016.174.5, , , , , ] 2021-09-17T10:12:30.054 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:30.072+0000 7f8b5457eb00 0 bucket checkpoint complete 2021-09-17T10:12:30.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:30.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:30.080 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:30.131 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:30.131 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:30.177 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:30.198+0000 7f0d0b442b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:30.177 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:30.198+0000 7f0d0b442b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:30.198+0000 7f0d0b442b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:30.198+0000 7f0d0b442b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:30.254+0000 7f0d0b442b00 1 bucket sync caught up with source: 2021-09-17T10:12:30.233 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, 00000000004.221.5, , 00000000004.297.5, , 00000000016.174.5, , , , , ] 2021-09-17T10:12:30.234 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, 00000000004.221.5, , 00000000004.297.5, , 00000000016.174.5, , , , , ] 2021-09-17T10:12:30.234 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:30.254+0000 7f0d0b442b00 0 bucket checkpoint complete 2021-09-17T10:12:30.263 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-18 zones={test-zone2, test-zone1} 2021-09-17T10:12:30.272 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:30.319 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:30.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:30.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:30.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:30.321 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:30.327 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:30.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:30.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:30.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:30.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:30.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:30.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:30.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:30.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:30.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:30.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:30.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:30.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:31.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:31.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:31.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:31.377 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-18 zones={test-zone2, test-zone1} 2021-09-17T10:12:31.378 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-18 zones={test-zone3, test-zone1} 2021-09-17T10:12:31.386 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:31.431 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:31.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:31.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:31.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:31.433 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:31.440 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:31.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:31.441 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:31.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:31.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:31.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:31.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:31.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:31.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:31.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:31.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:31.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:32.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:32.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:32.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:32.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:32.473 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-18 zones={test-zone3, test-zone1} 2021-09-17T10:12:32.474 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-18 zones={test-zone3, test-zone2} 2021-09-17T10:12:32.482 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:32.531 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:32.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:32.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:32.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:32.533 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:32.540 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:32.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-17T10:12:32.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-17T10:12:32.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-17T10:12:32.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:32.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:32.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:32.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:32.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:32.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-17T10:12:33.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:33.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:33.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-17T10:12:33.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:33.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:33.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-17T10:12:33.609 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-18 zones={test-zone3, test-zone2} 2021-09-17T10:12:33.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-09-17T10:12:33.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:12:33.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:12:33.652 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:12:33.697 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:33.697 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:33.793 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:12:33.819 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:12:33.819 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:12:33.819 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:12:33.820 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:12:33.820 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:33.820 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:12:33.820 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873553.669864_136.1", 2021-09-17T10:12:33.821 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:33.669864Z" 2021-09-17T10:12:33.821 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:33.821 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:12:33.821 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:12:33.821 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:12:33.822 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:33.822 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:12:33.822 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:12:33.822 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:12:33.823 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:33.823 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:12:33.824 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:12:33.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:12:33.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:12:33.826 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:12:33.871 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:33.871 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:33.966 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:12:33.966 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:12:33.966 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:12:33.967 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:12:33.967 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:12:33.967 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:12:33.967 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:12:33.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:12:33.968 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:12:33.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:12:33.968 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:12:33.969 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:12:33.969 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:12:33.969 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:12:33.969 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:12:33.969 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:12:33.970 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:12:33.970 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:12:33.970 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:12:33.970 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:12:33.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:12:33.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:12:33.971 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:12:33.971 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:12:33.971 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:12:33.972 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873553.669864_136.1", 2021-09-17T10:12:33.972 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:12:33.972 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:12:33.972 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:12:33.973 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:12:33.669864Z", 2021-09-17T10:12:33.973 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:12:33.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:12:33.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:12:33.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:12:33.974 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:12:33.974 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:12:33.974 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:12:33.975 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:12:33.975 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:12:33.975 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:12:33.975 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:12:33.976 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:12:33.976 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:12:33.976 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:12:33.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:12:33.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:12:33.977 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:12:33.977 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:12:33.977 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:12:33.977 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:12:33.978 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:12:33.978 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:12:33.978 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:12:33.978 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:12:33.979 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:12:33.980 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:12:33.980 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:12:33.981 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:12:33.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:12:33.981 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:12:33.981 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:12:33.982 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:12:33.982 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:12:33.982 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:12:33.983 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:12:33.984 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:12:33.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:12:33.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:12:33.986 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:12:34.027 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:34.027 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:34.137 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:12:34.137 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:12:34.138 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:12:34.138 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:12:34.138 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:12:34.138 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:12:34.139 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:12:34.139 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:34.139 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:12:34.139 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:12:34.140 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:12:34.140 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:12:34.140 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:12:34.140 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:12:34.140 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:12:34.141 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:12:34.141 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:12:34.141 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:12:34.141 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:12:34.142 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:34.142 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:34.142 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:12:34.142 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:12:34.143 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:12:34.143 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:12:34.143 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873553.648277_131.1", 2021-09-17T10:12:34.143 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:12:34.144 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:12:34.144 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:12:34.144 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:33.648277Z", 2021-09-17T10:12:34.144 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:12:34.145 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:34.145 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:34.145 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:12:34.145 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:12:34.146 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:12:34.146 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:12:34.146 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:12:34.146 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:12:34.146 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:12:34.147 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:12:34.147 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:12:34.147 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:12:34.147 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:34.148 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:34.148 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:12:34.148 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:12:34.148 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:12:34.148 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:12:34.149 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:12:34.149 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:12:34.149 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:12:34.149 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:12:34.150 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:12:34.150 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:12:34.150 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:34.150 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:34.151 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:12:34.151 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:34.151 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:12:34.151 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:12:34.151 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:12:34.152 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:34.152 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:12:34.153 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873553.669864_136.1 target=1_1631873553.648277_131.1 2021-09-17T10:12:39.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:12:39.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:12:39.156 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:12:39.202 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:39.202 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:39.306 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:12:39.306 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:12:39.307 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:12:39.307 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:12:39.307 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:12:39.307 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:12:39.308 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:12:39.308 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:39.308 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:12:39.308 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:12:39.309 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:12:39.309 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:12:39.309 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:12:39.309 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:12:39.310 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:12:39.310 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:12:39.310 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:12:39.310 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:12:39.310 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:12:39.311 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:39.311 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:39.311 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:12:39.311 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:12:39.312 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:12:39.312 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:12:39.312 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873553.669864_136.1", 2021-09-17T10:12:39.312 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:12:39.312 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:12:39.313 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:12:39.313 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:33.669864Z", 2021-09-17T10:12:39.313 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:12:39.313 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:39.314 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:39.314 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:12:39.314 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:12:39.314 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:12:39.314 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:12:39.315 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:12:39.315 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:12:39.315 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:12:39.315 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:12:39.316 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:12:39.316 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:12:39.316 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:39.316 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:39.316 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:12:39.317 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:12:39.317 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:12:39.317 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:12:39.317 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:12:39.318 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:12:39.318 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:12:39.318 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:12:39.318 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:12:39.318 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:12:39.319 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:39.319 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:39.319 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:12:39.319 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:12:39.320 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:12:39.320 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:12:39.320 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:12:39.320 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:12:39.320 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:12:39.322 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:12:39.322 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:12:39.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:12:39.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:12:39.324 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:12:39.369 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:39.370 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:39.478 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:12:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:12:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:12:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:12:39.479 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:12:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:12:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:12:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:12:39.480 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:12:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:12:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:12:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:12:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:12:39.481 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:12:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:12:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:12:39.482 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:12:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:12:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:12:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:12:39.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:12:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:12:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:12:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:12:39.484 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:12:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873553.669864_136.1", 2021-09-17T10:12:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:12:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:12:39.485 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:12:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:12:33.669864Z", 2021-09-17T10:12:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:12:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:12:39.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:12:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:12:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:12:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:12:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:12:39.487 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:12:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:12:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:12:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:12:39.488 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:12:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:12:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:12:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:12:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:12:39.489 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:12:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:12:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:12:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:12:39.490 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:12:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:12:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:12:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:12:39.491 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:12:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:12:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:12:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:12:39.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:12:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:12:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:12:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:12:39.493 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:12:39.494 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:12:39.495 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:12:39.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:39.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:39.544 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:39.593 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:39.594 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:39.639 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:39.662+0000 7fe6ee58cb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:39.639 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:39.662+0000 7fe6ee58cb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:39.639 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:39.662+0000 7fe6ee58cb00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:39.640 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:39.662+0000 7fe6ee58cb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:39.691 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:39.714+0000 7fe6ee58cb00 1 waiting for incremental sync to catch up: 2021-09-17T10:12:39.692 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , , , ] 2021-09-17T10:12:39.692 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000002.23.10, , , , , ] 2021-09-17T10:12:44.694 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:44.714+0000 7fe6ee58cb00 1 bucket sync caught up with source: 2021-09-17T10:12:44.694 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , 00000000006.31.9, , , , , ] 2021-09-17T10:12:44.695 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000002.23.10, , , , , ] 2021-09-17T10:12:44.695 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:44.714+0000 7fe6ee58cb00 0 bucket checkpoint complete 2021-09-17T10:12:44.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:44.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:44.727 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:44.777 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:44.777 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:44.822 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:44.840+0000 7f96e3836b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:44.823 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:44.840+0000 7f96e3836b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:44.823 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:44.840+0000 7f96e3836b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:44.823 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:44.840+0000 7f96e3836b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:44.874 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:44.892+0000 7f96e3836b00 1 bucket sync caught up with source: 2021-09-17T10:12:44.875 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000006.31.9, , , , , ] 2021-09-17T10:12:44.875 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000006.31.9, , , , , ] 2021-09-17T10:12:44.875 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:44.892+0000 7f96e3836b00 0 bucket checkpoint complete 2021-09-17T10:12:44.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:44.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:44.902 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:44.952 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:44.953 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:45.003 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:45.026+0000 7fb293e0fb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:45.004 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:45.026+0000 7fb293e0fb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:45.004 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:45.026+0000 7fb293e0fb00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:12:45.004 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:45.026+0000 7fb293e0fb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:45.082+0000 7fb293e0fb00 1 bucket sync caught up with source: 2021-09-17T10:12:45.061 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , 00000000006.31.9, , , , , ] 2021-09-17T10:12:45.061 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000006.31.9, , , , , ] 2021-09-17T10:12:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:45.082+0000 7fb293e0fb00 0 bucket checkpoint complete 2021-09-17T10:12:45.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:45.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:45.093 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:45.140 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:45.141 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:45.191 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.212+0000 7f19a6e91b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:45.191 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.212+0000 7f19a6e91b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:45.192 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.212+0000 7f19a6e91b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:45.192 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.212+0000 7f19a6e91b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:45.250 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.268+0000 7f19a6e91b00 1 bucket sync caught up with source: 2021-09-17T10:12:45.250 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000004.158.9, , , , , ] 2021-09-17T10:12:45.250 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000004.158.9, , , , , ] 2021-09-17T10:12:45.250 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.268+0000 7f19a6e91b00 0 bucket checkpoint complete 2021-09-17T10:12:45.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:45.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:45.280 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:45.330 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:45.330 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:45.378 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.396+0000 7ff9cc2ccb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:45.379 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.396+0000 7ff9cc2ccb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:45.379 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.396+0000 7ff9cc2ccb00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:45.379 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.396+0000 7ff9cc2ccb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:45.433 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.452+0000 7ff9cc2ccb00 1 bucket sync caught up with source: 2021-09-17T10:12:45.433 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000004.158.9, , , , , ] 2021-09-17T10:12:45.433 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000004.158.9, , , , , ] 2021-09-17T10:12:45.434 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.452+0000 7ff9cc2ccb00 0 bucket checkpoint complete 2021-09-17T10:12:45.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:45.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:45.463 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:45.511 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:45.511 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:45.558 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.576+0000 7f05e52b9b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:45.559 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.576+0000 7f05e52b9b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:45.559 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.576+0000 7f05e52b9b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:45.559 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.576+0000 7f05e52b9b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:45.611 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.632+0000 7f05e52b9b00 1 bucket sync caught up with source: 2021-09-17T10:12:45.611 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000004.177.9, , , , , ] 2021-09-17T10:12:45.612 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000004.177.9, , , , , ] 2021-09-17T10:12:45.612 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:45.632+0000 7f05e52b9b00 0 bucket checkpoint complete 2021-09-17T10:12:45.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:45.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:45.637 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:45.686 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:45.687 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:45.736 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:45.758+0000 7fbd596a3b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:45.736 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:45.758+0000 7fbd596a3b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:45.737 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:45.758+0000 7fbd596a3b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:45.737 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:45.758+0000 7fbd596a3b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:45.792 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:45.814+0000 7fbd596a3b00 1 bucket sync caught up with source: 2021-09-17T10:12:45.793 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , 00000000004.177.9, , , , , ] 2021-09-17T10:12:45.793 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000004.177.9, , , , , ] 2021-09-17T10:12:45.794 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:45.814+0000 7fbd596a3b00 0 bucket checkpoint complete 2021-09-17T10:12:45.824 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-19 zones={test-zone2, test-zone1} 2021-09-17T10:12:45.835 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:45.883 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:45.884 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:45.890 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:45.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:45.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:46.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:46.085 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-19 zones={test-zone2, test-zone1} 2021-09-17T10:12:46.086 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-19 zones={test-zone3, test-zone1} 2021-09-17T10:12:46.094 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:46.139 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:46.140 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:46.146 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:46.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:46.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:46.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:46.333 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-19 zones={test-zone3, test-zone1} 2021-09-17T10:12:46.334 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-19 zones={test-zone3, test-zone2} 2021-09-17T10:12:46.344 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:46.391 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:46.392 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:46.397 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:46.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:46.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:46.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:46.601 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-19 zones={test-zone3, test-zone2} 2021-09-17T10:12:48.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:48.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:48.108 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:48.157 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:48.157 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:48.202 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:48.222+0000 7ff0f9d56b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:48.203 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:48.222+0000 7ff0f9d56b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:48.203 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:48.222+0000 7ff0f9d56b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:48.203 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:48.222+0000 7ff0f9d56b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:48.252 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:48.274+0000 7ff0f9d56b00 1 waiting for incremental sync to catch up: 2021-09-17T10:12:48.253 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , 00000000006.31.9, , , 00000000050.124.11, , ] 2021-09-17T10:12:48.253 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000006.31.9, , , 00000000055.134.11, , ] 2021-09-17T10:12:53.256 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:53.277+0000 7ff0f9d56b00 1 bucket sync caught up with source: 2021-09-17T10:12:53.267 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , 00000000006.31.9, , , 00000000056.136.9, , ] 2021-09-17T10:12:53.267 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000006.31.9, , , 00000000055.134.11, , ] 2021-09-17T10:12:53.267 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:53.277+0000 7ff0f9d56b00 0 bucket checkpoint complete 2021-09-17T10:12:53.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:53.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:53.287 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:53.337 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:53.337 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:53.386 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.403+0000 7f0e5e847b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:53.386 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.403+0000 7f0e5e847b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:53.387 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.403+0000 7f0e5e847b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:53.387 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.403+0000 7f0e5e847b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:53.436 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.455+0000 7f0e5e847b00 1 bucket sync caught up with source: 2021-09-17T10:12:53.436 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000006.31.9, , , 00000000056.136.9, , ] 2021-09-17T10:12:53.437 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000006.31.9, , , 00000000056.136.9, , ] 2021-09-17T10:12:53.437 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.455+0000 7f0e5e847b00 0 bucket checkpoint complete 2021-09-17T10:12:53.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:53.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:53.463 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:53.585+0000 7f8e28121b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:53.585+0000 7f8e28121b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:53.585+0000 7f8e28121b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:12:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:53.585+0000 7f8e28121b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:53.621 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:53.641+0000 7f8e28121b00 1 bucket sync caught up with source: 2021-09-17T10:12:53.621 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , 00000000006.31.9, , , 00000000056.136.9, , ] 2021-09-17T10:12:53.622 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000006.31.9, , , 00000000056.136.9, , ] 2021-09-17T10:12:53.622 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:53.641+0000 7f8e28121b00 0 bucket checkpoint complete 2021-09-17T10:12:53.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:53.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:53.651 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:53.700 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:53.700 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:53.746 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.763+0000 7fdb6da80b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:53.747 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.763+0000 7fdb6da80b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:53.747 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.763+0000 7fdb6da80b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:53.747 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.763+0000 7fdb6da80b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:53.801 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.819+0000 7fdb6da80b00 1 bucket sync caught up with source: 2021-09-17T10:12:53.802 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000004.158.9, , , 00000000060.327.9, , ] 2021-09-17T10:12:53.802 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000004.158.9, , , 00000000060.327.9, , ] 2021-09-17T10:12:53.802 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.819+0000 7fdb6da80b00 0 bucket checkpoint complete 2021-09-17T10:12:53.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:53.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:53.832 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:53.878 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:53.879 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:53.925 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.943+0000 7f028205eb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:53.926 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.943+0000 7f028205eb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:53.926 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.943+0000 7f028205eb00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:12:53.926 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.943+0000 7f028205eb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:53.978 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.995+0000 7f028205eb00 1 bucket sync caught up with source: 2021-09-17T10:12:53.979 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000004.158.9, , , 00000000060.327.9, , ] 2021-09-17T10:12:53.979 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000004.158.9, , , 00000000060.327.9, , ] 2021-09-17T10:12:53.979 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:53.995+0000 7f028205eb00 0 bucket checkpoint complete 2021-09-17T10:12:54.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:54.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:54.009 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:54.056 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:54.056 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:54.100 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:54.119+0000 7f493c26ab00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:54.100 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:54.119+0000 7f493c26ab00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:54.100 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:54.119+0000 7f493c26ab00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:12:54.100 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:54.119+0000 7f493c26ab00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:54.151 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:54.167+0000 7f493c26ab00 1 bucket sync caught up with source: 2021-09-17T10:12:54.151 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , 00000000004.177.9, , , 00000000057.341.5, , ] 2021-09-17T10:12:54.152 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , 00000000004.177.9, , , 00000000057.341.5, , ] 2021-09-17T10:12:54.152 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:12:54.167+0000 7f493c26ab00 0 bucket checkpoint complete 2021-09-17T10:12:54.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:54.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:12:54.178 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:12:54.228 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:12:54.228 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:12:54.281 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:54.301+0000 7f2fa2f5ab00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:12:54.282 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:54.301+0000 7f2fa2f5ab00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:12:54.282 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:54.301+0000 7f2fa2f5ab00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:12:54.282 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:54.301+0000 7f2fa2f5ab00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:12:54.337 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:54.357+0000 7f2fa2f5ab00 1 bucket sync caught up with source: 2021-09-17T10:12:54.338 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , 00000000004.177.9, , , 00000000057.341.5, , ] 2021-09-17T10:12:54.338 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , 00000000004.177.9, , , 00000000057.341.5, , ] 2021-09-17T10:12:54.338 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:12:54.357+0000 7f2fa2f5ab00 0 bucket checkpoint complete 2021-09-17T10:12:54.367 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-19 zones={test-zone2, test-zone1} 2021-09-17T10:12:54.377 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:12:54.387 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:54.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.400 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:12:54.412 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:12:54.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.417 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.418 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.421 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:12:54.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:54.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:54.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:12:54.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:54.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:54.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:54.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:54.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:55.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:55.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:55.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:55.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:55.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:55.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:55.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:55.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:55.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:55.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:56.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:56.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:56.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:56.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:56.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:56.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:56.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:56.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:56.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:56.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:56.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:57.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:57.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:57.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:57.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:57.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:57.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:57.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:57.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:57.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:57.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:58.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:58.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:58.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:58.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:58.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:58.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:58.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:58.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:58.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:58.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:58.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:59.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:59.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:59.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:59.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:59.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:59.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:59.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:59.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:59.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:12:59.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:00.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:00.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:00.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:00.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:00.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:00.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:00.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:00.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:00.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:00.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:00.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:01.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:01.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:01.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:01.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:01.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:01.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:01.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:01.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:01.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:01.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:02.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:02.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:02.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:02.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:02.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:02.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:02.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:02.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:02.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:02.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:02.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:03.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:03.200 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-19 zones={test-zone2, test-zone1} 2021-09-17T10:13:03.200 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-19 zones={test-zone3, test-zone1} 2021-09-17T10:13:03.207 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:13:03.213 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:13:03.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.227 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:13:03.236 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:13:03.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:03.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:13:03.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:13:03.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:13:03.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:03.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:03.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:03.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:03.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:03.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:03.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:04.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:04.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:04.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:04.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:04.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:04.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:04.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:04.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:04.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:04.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:04.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:05.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:05.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:05.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:05.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:05.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:05.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:05.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:05.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:05.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:05.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:05.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:06.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:06.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:06.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:06.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:06.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:06.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:06.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:06.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:06.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:06.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:07.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:07.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:07.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:07.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:07.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:07.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:07.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:07.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:07.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:07.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:07.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:08.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:08.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:08.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:08.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:08.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:08.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:08.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:08.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:08.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:08.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:08.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:09.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:09.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:09.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:09.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:09.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:09.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:09.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:09.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:09.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:09.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:10.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:10.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:10.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:10.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:10.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:10.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:10.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:10.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:10.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:10.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:11.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:11.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:11.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:11.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:11.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:11.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:11.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:11.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:11.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:11.867 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-19 zones={test-zone3, test-zone1} 2021-09-17T10:13:11.867 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-19 zones={test-zone3, test-zone2} 2021-09-17T10:13:11.878 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:13:11.928 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:13:11.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.942 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:13:11.954 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-17T10:13:11.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:11.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:13:11.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:13:12.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-17T10:13:12.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:12.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:12.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:12.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:12.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:12.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:12.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:12.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:12.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:12.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:13.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:13.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:13.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:13.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:13.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:13.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:13.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:13.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:13.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:13.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:13.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:14.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:14.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:14.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:14.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:14.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:14.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:14.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:14.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:14.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:14.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:15.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:15.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:15.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:15.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:15.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:15.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:15.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:15.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:15.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:15.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:15.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:16.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:16.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:16.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:16.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:16.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:16.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:16.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:16.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:16.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:16.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:16.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:17.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:17.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:17.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:17.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:17.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:17.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:17.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:17.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:17.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:17.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:18.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:18.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:18.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:18.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:18.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:18.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:18.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:18.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:18.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:18.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:19.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:19.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:19.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:19.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:19.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:19.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:19.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:19.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:19.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:19.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:19.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:20.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:20.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:20.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:20.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:20.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:20.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:20.642 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-19 zones={test-zone3, test-zone2} 2021-09-17T10:13:20.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:20.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:20.644 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:20.686 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:20.687 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:20.821 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:13:20.821 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:20.822 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:20.822 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:57.694090Z" 2021-09-17T10:13:20.822 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:20.822 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:20.822 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:20.823 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:57.696845Z" 2021-09-17T10:13:20.823 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:20.823 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:20.823 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:20.823 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:20.824 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:20.824 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:20.824 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:20.824 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:35.690760Z" 2021-09-17T10:13:20.825 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:20.825 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:13:20.826 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-09-17T10:13:20.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:20.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:20.828 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:20.871 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:20.872 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:20.997 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:20.998 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:20.998 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:20.998 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:20.999 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:20.999 INFO:teuthology.orchestra.run.smithi050.stdout: "instance_id": 16292831388237778446 2021-09-17T10:13:20.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:20.999 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:20.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:21.000 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:21.000 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:21.000 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "incremental-sync", 2021-09-17T10:13:21.000 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:21.001 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:21.001 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:21.001 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:21.001 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:12:57.694090Z" 2021-09-17T10:13:21.002 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:21.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:21.002 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:21.003 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:21.003 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "incremental-sync", 2021-09-17T10:13:21.003 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:21.003 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:21.003 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:21.004 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:21.004 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:12:57.696845Z" 2021-09-17T10:13:21.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:21.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:21.005 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:21.005 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:21.005 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "incremental-sync", 2021-09-17T10:13:21.005 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:21.006 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:21.006 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:21.006 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:21.006 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000" 2021-09-17T10:13:21.007 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:21.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:21.007 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:21.007 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:21.008 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "incremental-sync", 2021-09-17T10:13:21.008 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:21.008 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:21.008 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:21.008 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:21.009 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:12:35.690760Z" 2021-09-17T10:13:21.009 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.009 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.009 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:21.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:21.010 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:21.010 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 0, 2021-09-17T10:13:21.010 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 0 2021-09-17T10:13:21.010 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.011 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:21.012 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-09-17T10:13:21.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:21.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:21.013 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:21.054 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:21.054 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:21.182 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:13:21.182 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:21.182 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:21.183 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:57.694090Z" 2021-09-17T10:13:21.183 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:21.183 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:21.183 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:21.184 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:57.696845Z" 2021-09-17T10:13:21.184 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:21.184 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:21.184 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:21.184 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:21.185 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:21.185 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:21.185 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:21.185 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:35.690760Z" 2021-09-17T10:13:21.185 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:21.186 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:13:21.187 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-09-17T10:13:21.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:21.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:21.188 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:21.270 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:21.271 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:21.391 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:13:21.391 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:13:21.391 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:13:21.392 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:13:21.392 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:13:21.392 INFO:teuthology.orchestra.run.smithi018.stdout: "instance_id": 14795371724412548584 2021-09-17T10:13:21.392 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:21.392 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:13:21.393 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:21.393 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:13:21.393 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:21.393 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:21.394 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:21.394 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:21.394 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:21.394 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:21.395 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:57.694090Z" 2021-09-17T10:13:21.395 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:21.395 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:21.395 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:21.395 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:13:21.396 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:21.396 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:21.396 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:21.396 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:21.396 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:21.397 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:21.397 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:57.696845Z" 2021-09-17T10:13:21.397 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:21.397 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:21.397 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:21.398 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:13:21.398 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:21.398 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:21.398 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:21.398 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:21.399 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:21.399 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:21.399 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000" 2021-09-17T10:13:21.399 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:21.400 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:21.400 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:21.400 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:13:21.400 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:21.400 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:21.401 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:21.401 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:21.401 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:21.401 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:21.401 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:35.690760Z" 2021-09-17T10:13:21.402 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:21.402 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:21.402 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:13:21.402 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:21.403 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:13:21.403 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 0, 2021-09-17T10:13:21.403 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 0 2021-09-17T10:13:21.403 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:21.403 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:13:21.404 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-09-17T10:13:21.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:21.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:21.406 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:21.449 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:21.449 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:21.577 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:13:21.577 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:21.577 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:21.578 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:57.694090Z" 2021-09-17T10:13:21.578 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:21.578 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:21.579 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:21.579 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:57.696845Z" 2021-09-17T10:13:21.579 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:21.579 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:21.579 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:21.580 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:21.580 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:21.580 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:21.580 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:21.581 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:35.690760Z" 2021-09-17T10:13:21.581 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:21.581 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:13:21.582 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-09-17T10:13:21.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:21.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:21.584 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:21.630 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:21.630 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:21.754 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:21.755 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:21.755 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:21.755 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:21.756 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:21.756 INFO:teuthology.orchestra.run.smithi050.stdout: "instance_id": 6505317987491692740 2021-09-17T10:13:21.756 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:21.756 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:21.756 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:21.757 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:21.757 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:21.757 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "incremental-sync", 2021-09-17T10:13:21.758 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:21.758 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:21.758 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:21.758 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:21.759 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:12:57.694090Z" 2021-09-17T10:13:21.759 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.759 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:21.759 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:21.760 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:21.760 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:21.760 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "incremental-sync", 2021-09-17T10:13:21.760 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:21.760 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:12:57.696845Z" 2021-09-17T10:13:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:21.762 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:21.762 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:21.762 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:21.762 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "incremental-sync", 2021-09-17T10:13:21.763 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:21.763 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:21.763 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:21.763 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:21.765 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000" 2021-09-17T10:13:21.765 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.765 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:21.765 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:21.766 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:21.766 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:21.766 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "incremental-sync", 2021-09-17T10:13:21.766 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:21.766 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:21.767 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:21.767 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:21.767 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:12:35.690760Z" 2021-09-17T10:13:21.768 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.768 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.768 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:21.768 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:21.769 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:21.769 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 0, 2021-09-17T10:13:21.769 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 0 2021-09-17T10:13:21.769 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.769 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:21.771 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-09-17T10:13:21.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:21.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:21.772 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:21.815 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:21.816 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:21.946 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2021-09-17T10:13:21.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:21.947 INFO:teuthology.orchestra.run.smithi050.stdout: "last_update": "2021-09-17T10:12:57.764637Z" 2021-09-17T10:13:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:21.948 INFO:teuthology.orchestra.run.smithi050.stdout: "last_update": "2021-09-17T10:12:57.767565Z" 2021-09-17T10:13:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: "last_update": "0.000000" 2021-09-17T10:13:21.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: "last_update": "2021-09-17T10:12:35.761162Z" 2021-09-17T10:13:21.950 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:21.951 INFO:teuthology.orchestra.run.smithi050.stdout:] 2021-09-17T10:13:21.952 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-09-17T10:13:21.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:21.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:21.953 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:21.999 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:22.000 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:22.119 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:13:22.120 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:13:22.120 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:13:22.120 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:13:22.121 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:13:22.121 INFO:teuthology.orchestra.run.smithi018.stdout: "instance_id": 4951927267074774482 2021-09-17T10:13:22.121 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.121 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:13:22.121 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.122 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:13:22.122 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:22.122 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:22.122 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:22.123 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:22.123 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:22.123 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:22.123 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:57.764637Z" 2021-09-17T10:13:22.123 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.124 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.124 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.124 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:13:22.124 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:22.124 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:22.125 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:22.125 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:22.125 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:22.125 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:22.126 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:57.767565Z" 2021-09-17T10:13:22.126 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.126 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.126 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.126 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:13:22.127 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:22.127 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:22.127 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:22.127 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:22.127 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:22.128 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:22.128 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000" 2021-09-17T10:13:22.128 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.128 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.129 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.129 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:13:22.129 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:22.129 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:22.129 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:22.130 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:22.130 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:22.130 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:22.130 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:35.761162Z" 2021-09-17T10:13:22.130 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.131 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.131 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:13:22.131 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.131 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:13:22.131 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 0, 2021-09-17T10:13:22.132 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 0 2021-09-17T10:13:22.132 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.132 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:13:22.133 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-09-17T10:13:22.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:22.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:22.135 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:22.177 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:22.178 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:22.308 INFO:teuthology.orchestra.run.smithi050.stdout:[ 2021-09-17T10:13:22.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:22.309 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:22.309 INFO:teuthology.orchestra.run.smithi050.stdout: "last_update": "2021-09-17T10:12:57.764637Z" 2021-09-17T10:13:22.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:22.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:22.309 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:22.310 INFO:teuthology.orchestra.run.smithi050.stdout: "last_update": "2021-09-17T10:12:57.767565Z" 2021-09-17T10:13:22.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:22.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:22.310 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:22.311 INFO:teuthology.orchestra.run.smithi050.stdout: "last_update": "0.000000" 2021-09-17T10:13:22.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:22.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:22.311 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:22.312 INFO:teuthology.orchestra.run.smithi050.stdout: "last_update": "2021-09-17T10:12:35.761162Z" 2021-09-17T10:13:22.312 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:22.312 INFO:teuthology.orchestra.run.smithi050.stdout:] 2021-09-17T10:13:22.313 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-09-17T10:13:22.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:22.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:22.315 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:22.358 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:22.358 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:22.476 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:13:22.476 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:13:22.476 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:13:22.476 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:13:22.477 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:13:22.477 INFO:teuthology.orchestra.run.smithi018.stdout: "instance_id": 13422549656113866042 2021-09-17T10:13:22.477 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.477 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:13:22.478 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.478 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:13:22.478 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:22.478 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:22.479 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:22.479 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:22.479 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:22.479 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:22.480 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:57.764637Z" 2021-09-17T10:13:22.480 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.480 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.480 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.480 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:13:22.481 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:22.481 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:22.481 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:22.481 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:22.482 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:22.482 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:22.482 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:57.767565Z" 2021-09-17T10:13:22.482 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.483 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.483 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.483 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:13:22.483 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:22.484 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:22.484 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:22.484 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:22.484 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:22.485 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:22.485 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000" 2021-09-17T10:13:22.485 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.485 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.486 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.486 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:13:22.486 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:22.486 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:22.486 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:22.487 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:22.487 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:22.487 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:22.487 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:35.761162Z" 2021-09-17T10:13:22.488 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.488 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.488 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:13:22.488 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.488 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:13:22.489 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 0, 2021-09-17T10:13:22.489 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 0 2021-09-17T10:13:22.489 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.489 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:13:22.490 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-09-17T10:13:22.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:22.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:22.492 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:22.534 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:22.534 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:22.665 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:13:22.666 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.666 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:22.666 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:57.655582Z" 2021-09-17T10:13:22.666 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.667 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.667 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:22.667 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:57.658893Z" 2021-09-17T10:13:22.667 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.668 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.668 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:22.668 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:22.668 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.668 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.669 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:22.669 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:35.651931Z" 2021-09-17T10:13:22.669 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.669 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:13:22.670 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-09-17T10:13:22.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:22.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:22.672 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:22.715 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:22.715 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:22.836 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:13:22.836 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:13:22.836 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:13:22.837 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:13:22.837 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:13:22.837 INFO:teuthology.orchestra.run.smithi018.stdout: "instance_id": 4991463302324557773 2021-09-17T10:13:22.837 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.837 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:13:22.838 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.838 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:13:22.838 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:22.838 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:22.839 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:22.839 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:22.839 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:22.839 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:22.840 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:57.655582Z" 2021-09-17T10:13:22.840 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.840 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.840 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.841 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:13:22.841 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:22.841 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:22.841 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:22.841 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:22.842 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:22.842 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:22.842 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:57.658893Z" 2021-09-17T10:13:22.843 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.843 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.843 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.843 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:13:22.844 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:22.844 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:22.844 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:22.844 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:22.845 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:22.845 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:22.845 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000" 2021-09-17T10:13:22.845 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.845 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.846 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:22.846 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:13:22.846 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:22.846 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "incremental-sync", 2021-09-17T10:13:22.847 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:22.847 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:22.847 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:22.847 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:22.847 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:12:35.651931Z" 2021-09-17T10:13:22.848 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.848 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.848 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:13:22.848 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:22.848 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:13:22.849 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 0, 2021-09-17T10:13:22.849 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 0 2021-09-17T10:13:22.849 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:22.849 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:13:22.851 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-09-17T10:13:22.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:22.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:22.852 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:22.931 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:22.932 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:23.065 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:13:23.065 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:23.066 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:23.066 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:57.655582Z" 2021-09-17T10:13:23.066 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:23.066 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:23.066 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:23.067 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:57.658893Z" 2021-09-17T10:13:23.067 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:23.067 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:23.067 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:23.068 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:23.068 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:23.068 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:23.068 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:23.069 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:12:35.651931Z" 2021-09-17T10:13:23.069 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:23.069 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:13:23.070 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-09-17T10:13:23.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:23.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:23.072 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:23.115 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:23.116 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:23.239 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:23.240 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:23.240 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:23.240 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:23.240 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:23.241 INFO:teuthology.orchestra.run.smithi050.stdout: "instance_id": 8715768237370895338 2021-09-17T10:13:23.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:23.241 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:23.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:23.241 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:23.242 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:23.242 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "incremental-sync", 2021-09-17T10:13:23.242 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-17T10:13:23.242 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:23.243 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:23.243 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:23.243 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:12:57.655582Z" 2021-09-17T10:13:23.243 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:23.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:23.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:23.244 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:23.244 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:23.244 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "incremental-sync", 2021-09-17T10:13:23.244 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000004369", 2021-09-17T10:13:23.245 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:23.245 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:23.245 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:23.245 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:12:57.658893Z" 2021-09-17T10:13:23.245 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:23.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:23.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:23.246 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:23.247 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:23.247 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "incremental-sync", 2021-09-17T10:13:23.247 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:23.247 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:23.248 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:23.248 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:23.248 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000" 2021-09-17T10:13:23.248 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:23.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:23.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:23.249 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:23.249 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:23.249 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "incremental-sync", 2021-09-17T10:13:23.250 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "00000000000000000000:00000000000000002374", 2021-09-17T10:13:23.250 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:23.250 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:23.250 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:23.251 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:12:35.651931Z" 2021-09-17T10:13:23.251 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:23.251 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:23.251 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:23.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:23.252 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:23.252 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 0, 2021-09-17T10:13:23.252 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 0 2021-09-17T10:13:23.252 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:23.253 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:23.254 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-09-17T10:13:23.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-09-17T10:13:23.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:23.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:23.285 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:23.329 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:23.330 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:23.400 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:13:23.426 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:13:23.427 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:23.427 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:23.427 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:13:23.427 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:23.427 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:23.428 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873603.301112_147.1", 2021-09-17T10:13:23.428 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:13:23.301112Z" 2021-09-17T10:13:23.428 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:23.428 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:23.429 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:23.429 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:23.429 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:23.429 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:23.430 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:23.430 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:23.430 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:23.430 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:13:23.432 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:13:23.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:23.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:23.433 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:23.476 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:23.588 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:23.588 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:23.588 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:23.589 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:23.589 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:23.589 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:23.589 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:23.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:23.590 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:23.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:23.590 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:23.590 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:23.591 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:23.591 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:23.591 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:23.591 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:13:23.592 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:23.592 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:23.592 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:23.592 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:23.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:23.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:23.593 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:23.593 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:23.593 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:23.593 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873603.289881_144.1", 2021-09-17T10:13:23.594 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:23.594 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:23.594 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:23.594 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:13:23.289881Z", 2021-09-17T10:13:23.595 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:23.595 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:23.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:23.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:23.597 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:23.597 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:23.597 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:23.597 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:23.598 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:23.598 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:23.598 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:23.598 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:23.599 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:23.599 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:23.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:23.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:23.599 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:23.600 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:23.600 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:23.600 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:23.600 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:23.601 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:23.601 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:23.601 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:23.601 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:23.601 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:23.602 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:23.602 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:23.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:23.602 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:23.603 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:13:23.603 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:13:23.603 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:23.603 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:23.605 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631873603.301112_147.1 target=1_1631873603.289881_144.1 2021-09-17T10:13:28.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:28.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:28.607 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:28.650 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:28.650 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:28.770 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:28.771 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:28.771 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:28.771 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:28.771 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:28.771 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:28.772 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:28.772 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:28.772 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:28.772 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:28.773 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:28.773 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:28.773 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:28.773 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:28.773 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:28.774 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:13:28.774 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:28.774 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:28.774 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:28.775 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:28.775 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:28.775 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:28.775 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:28.775 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:28.776 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:28.776 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873603.301112_147.1", 2021-09-17T10:13:28.776 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:28.776 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:28.777 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:28.777 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:13:23.301112Z", 2021-09-17T10:13:28.777 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:28.777 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:28.777 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:28.778 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:28.778 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:28.778 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:28.778 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:28.779 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:28.779 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:28.779 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:28.779 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:28.779 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:28.780 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:28.780 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:28.780 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:28.780 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:28.780 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:28.781 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:28.781 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:28.781 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:28.781 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:28.782 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:28.782 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:28.782 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:28.782 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:28.782 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:28.783 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:28.783 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:28.783 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:28.783 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:28.783 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:13:28.784 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:13:28.784 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:28.784 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:28.785 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:13:28.786 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:13:28.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:28.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:28.787 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:28.828 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:28.828 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:28.945 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:13:28.945 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:13:28.946 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:13:28.946 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:13:28.946 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:13:28.947 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:28.947 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:28.947 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:28.947 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:13:28.947 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:28.948 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:13:28.948 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:28.948 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:28.948 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:28.949 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:28.949 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:13:28.949 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:28.949 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:28.950 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:28.950 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:28.950 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:28.950 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:28.950 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:13:28.951 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:28.951 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:28.951 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873603.301112_147.1", 2021-09-17T10:13:28.951 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:28.952 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:28.952 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:28.952 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:13:23.301112Z", 2021-09-17T10:13:28.952 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:28.952 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:28.953 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:28.953 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:28.953 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:13:28.953 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:28.953 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:28.954 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:28.954 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:28.954 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:28.954 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:28.955 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:13:28.955 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:28.955 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:28.955 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:28.955 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:28.956 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:13:28.956 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:28.956 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:28.956 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:28.957 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:28.957 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:28.957 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:28.957 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:13:28.957 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:28.958 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:28.958 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:28.958 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:13:28.958 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:28.958 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:13:28.959 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:13:28.959 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:13:28.959 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:28.959 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:13:28.960 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:13:28.961 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:13:28.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:28.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:28.964 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:29.012 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:29.013 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:29.129 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:29.129 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:29.129 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:29.130 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:29.130 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:29.130 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:29.130 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:29.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:29.131 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:29.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:29.131 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:29.131 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:29.132 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:29.132 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:29.132 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:29.132 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:13:29.133 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:29.133 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:29.133 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:29.133 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:29.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:29.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:29.134 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:29.134 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:29.134 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:29.135 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873603.301112_147.1", 2021-09-17T10:13:29.135 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:29.135 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:29.135 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:29.136 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:13:23.301112Z", 2021-09-17T10:13:29.136 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:29.136 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:29.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:29.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:29.137 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:29.137 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:29.138 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:29.138 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:29.138 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:29.139 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:29.139 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:29.139 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:29.139 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:29.140 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:29.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:29.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:29.140 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:29.140 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:29.141 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:29.141 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:29.141 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:29.141 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:29.141 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:29.142 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:29.142 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:29.142 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:29.142 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:29.142 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:29.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:29.143 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:29.143 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:13:29.143 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:13:29.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:29.144 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:29.145 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:13:29.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:29.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:29.196 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:29.245 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:29.245 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:29.293 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:29.309+0000 7fc3908d5b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:29.294 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:29.309+0000 7fc3908d5b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:29.294 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:29.309+0000 7fc3908d5b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:13:29.294 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:29.309+0000 7fc3908d5b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:29.349 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:29.365+0000 7fc3908d5b00 1 waiting for incremental sync to catch up: 2021-09-17T10:13:29.350 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , , , ] 2021-09-17T10:13:29.350 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000001.28.5, , ] 2021-09-17T10:13:34.352 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:34.372+0000 7fc3908d5b00 1 bucket sync caught up with source: 2021-09-17T10:13:34.353 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000001.28.5, , ] 2021-09-17T10:13:34.353 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000001.28.5, , ] 2021-09-17T10:13:34.353 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:34.372+0000 7fc3908d5b00 0 bucket checkpoint complete 2021-09-17T10:13:34.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:34.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:34.383 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:34.432 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:34.433 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:34.483 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:34.497+0000 7f2603d57b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:34.484 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:34.497+0000 7f2603d57b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:34.484 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:34.497+0000 7f2603d57b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:13:34.484 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:34.497+0000 7f2603d57b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:34.538 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:34.553+0000 7f2603d57b00 1 bucket sync caught up with source: 2021-09-17T10:13:34.538 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000001.28.5, , ] 2021-09-17T10:13:34.538 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000001.28.5, , ] 2021-09-17T10:13:34.539 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:34.553+0000 7f2603d57b00 0 bucket checkpoint complete 2021-09-17T10:13:34.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:34.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:34.566 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:34.616 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:34.616 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:34.665 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:34.684+0000 7f1c0856cb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:34.666 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:34.684+0000 7f1c0856cb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:34.666 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:34.684+0000 7f1c0856cb00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:13:34.666 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:34.684+0000 7f1c0856cb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:34.723 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:34.744+0000 7f1c0856cb00 1 bucket sync caught up with source: 2021-09-17T10:13:34.724 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000001.28.5, , ] 2021-09-17T10:13:34.724 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000001.28.5, , ] 2021-09-17T10:13:34.724 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:34.744+0000 7f1c0856cb00 0 bucket checkpoint complete 2021-09-17T10:13:34.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:34.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:34.755 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:34.807 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:34.807 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:34.852 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:34.865+0000 7fdafc94eb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:34.852 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:34.865+0000 7fdafc94eb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:34.852 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:34.865+0000 7fdafc94eb00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:13:34.852 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:34.865+0000 7fdafc94eb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:34.904 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:34.921+0000 7fdafc94eb00 1 bucket sync caught up with source: 2021-09-17T10:13:34.905 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000001.402.5, , ] 2021-09-17T10:13:34.905 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000001.402.5, , ] 2021-09-17T10:13:34.905 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:34.921+0000 7fdafc94eb00 0 bucket checkpoint complete 2021-09-17T10:13:34.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:34.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:34.934 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:34.983 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:34.983 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:35.033 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:35.049+0000 7f46dce73b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:35.034 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:35.049+0000 7f46dce73b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:35.034 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:35.049+0000 7f46dce73b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:13:35.034 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:35.049+0000 7f46dce73b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:35.088 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:35.101+0000 7f46dce73b00 1 bucket sync caught up with source: 2021-09-17T10:13:35.089 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000001.402.5, , ] 2021-09-17T10:13:35.089 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000001.402.5, , ] 2021-09-17T10:13:35.089 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:35.101+0000 7f46dce73b00 0 bucket checkpoint complete 2021-09-17T10:13:35.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:35.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:35.114 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:35.161 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:35.161 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:35.210 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:35.225+0000 7f54cc667b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:35.211 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:35.225+0000 7f54cc667b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:35.211 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:35.225+0000 7f54cc667b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:13:35.211 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:35.225+0000 7f54cc667b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:35.260 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:35.273+0000 7f54cc667b00 1 bucket sync caught up with source: 2021-09-17T10:13:35.260 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000001.399.5, , ] 2021-09-17T10:13:35.261 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000001.399.5, , ] 2021-09-17T10:13:35.261 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:35.273+0000 7f54cc667b00 0 bucket checkpoint complete 2021-09-17T10:13:35.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:35.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:35.288 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:35.338 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:35.339 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:35.381 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:35.400+0000 7fc115b6eb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:35.381 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:35.400+0000 7fc115b6eb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:35.382 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:35.400+0000 7fc115b6eb00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:13:35.382 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:35.400+0000 7fc115b6eb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:35.452+0000 7fc115b6eb00 1 bucket sync caught up with source: 2021-09-17T10:13:35.432 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000001.399.5, , ] 2021-09-17T10:13:35.432 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000001.399.5, , ] 2021-09-17T10:13:35.432 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:35.452+0000 7fc115b6eb00 0 bucket checkpoint complete 2021-09-17T10:13:35.464 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-20 zones={test-zone2, test-zone1} 2021-09-17T10:13:35.473 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:13:35.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:35.519 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:13:35.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:35.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:35.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:35.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:35.717 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-20 zones={test-zone2, test-zone1} 2021-09-17T10:13:35.718 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-20 zones={test-zone3, test-zone1} 2021-09-17T10:13:35.727 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:13:35.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:35.779 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:13:35.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:35.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:35.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:35.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:35.981 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-20 zones={test-zone3, test-zone1} 2021-09-17T10:13:35.982 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-20 zones={test-zone3, test-zone2} 2021-09-17T10:13:35.991 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:13:36.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:36.039 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:13:36.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:36.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:36.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:36.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:36.245 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-20 zones={test-zone3, test-zone2} 2021-09-17T10:13:36.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:36.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:36.257 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:36.302 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:36.302 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:36.379 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:13:36.398 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:13:36.398 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:36.399 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:36.399 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:13:36.399 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:36.399 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:36.400 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873616.274233_179.1", 2021-09-17T10:13:36.400 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:13:36.274233Z" 2021-09-17T10:13:36.400 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:36.400 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:36.401 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:36.401 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:36.401 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:36.401 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:36.401 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:36.402 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:36.402 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:36.403 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:13:36.404 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:13:36.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:36.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:36.406 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:36.443 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:36.444 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:36.550 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:36.551 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:36.551 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:36.552 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:36.552 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:36.552 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:36.552 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:36.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:36.553 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:36.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:36.553 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:36.554 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:36.554 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:36.554 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:36.554 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:36.555 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:13:36.555 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:36.555 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:36.555 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:36.556 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:36.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:36.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:36.556 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:36.557 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:36.557 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:36.557 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873616.274233_179.1", 2021-09-17T10:13:36.557 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:36.557 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:36.558 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:36.558 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:13:36.274233Z", 2021-09-17T10:13:36.558 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:36.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:36.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:36.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:36.559 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:36.559 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:36.559 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:36.560 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:36.560 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:36.560 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:36.560 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:36.560 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:36.561 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:36.561 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:36.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:36.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:36.562 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:36.562 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:36.562 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:36.562 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:36.562 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:36.563 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:36.563 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:36.563 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:36.563 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:36.564 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:36.564 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:36.564 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:36.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:36.565 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:36.565 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:13:36.565 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:13:36.565 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:36.565 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:36.567 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:13:36.567 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:13:36.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:36.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:36.569 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:36.613 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:36.614 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:36.737 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:13:36.738 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:13:36.738 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:13:36.738 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:13:36.738 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:13:36.739 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:36.739 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:36.739 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:36.739 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:13:36.739 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:36.740 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:13:36.740 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:36.740 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:36.740 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:36.741 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:36.741 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:13:36.741 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:36.741 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:36.742 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:36.742 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:36.742 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:36.742 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:36.742 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:13:36.743 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:36.743 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:36.743 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873616.274233_179.1", 2021-09-17T10:13:36.743 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:36.744 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:36.744 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:36.744 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:13:36.274233Z", 2021-09-17T10:13:36.744 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:36.745 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:36.745 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:36.745 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:36.745 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:13:36.746 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:36.746 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:36.746 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:36.746 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:36.746 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:36.747 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:36.747 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:13:36.747 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:36.747 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:36.748 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:36.748 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:36.748 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:13:36.748 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:36.748 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:36.749 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:36.749 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:36.749 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:36.749 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:36.750 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:13:36.750 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:36.750 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:36.750 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:36.750 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:13:36.751 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:36.751 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:13:36.751 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:13:36.751 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:13:36.752 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:36.752 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:13:36.753 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:13:36.754 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:13:36.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:36.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:36.755 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:36.798 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:36.798 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:36.917 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:13:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873603.301112_147.1", 2021-09-17T10:13:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:13:23.301112Z", 2021-09-17T10:13:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:13:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:13:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:36.931 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:36.933 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1631873616.274233_179.1 target=1_1631873603.301112_147.1 2021-09-17T10:13:41.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:41.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:41.936 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:41.978 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:41.979 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:42.097 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:42.098 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:42.098 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:42.098 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:42.098 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:42.099 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:42.099 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:42.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:42.099 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:42.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:42.100 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:42.100 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:42.100 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:42.100 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:42.101 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:42.101 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:13:42.101 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:42.102 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:42.102 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:42.102 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:42.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:42.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:42.103 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:42.103 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:42.103 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:42.103 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873616.274233_179.1", 2021-09-17T10:13:42.104 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:42.104 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:42.104 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:42.104 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:13:36.274233Z", 2021-09-17T10:13:42.105 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:42.105 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:42.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:42.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:42.105 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:42.106 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:42.106 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:42.106 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:42.106 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:42.107 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:42.107 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:42.107 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:42.108 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:42.108 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:42.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:42.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:42.108 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:42.109 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:42.109 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:42.109 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:42.109 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:42.110 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:42.110 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:42.110 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:42.110 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:42.111 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:42.111 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:42.111 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:42.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:42.112 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:42.112 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:13:42.112 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:13:42.112 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:42.112 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:42.114 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:13:42.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:42.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:42.164 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:42.262 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:42.280+0000 7f5f5b09cb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:42.280+0000 7f5f5b09cb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:42.280+0000 7f5f5b09cb00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:13:42.263 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:42.280+0000 7f5f5b09cb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:42.336+0000 7f5f5b09cb00 1 waiting for incremental sync to catch up: 2021-09-17T10:13:42.317 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000001.28.5, , ] 2021-09-17T10:13:42.317 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000003.31.12, , ] 2021-09-17T10:13:47.320 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:47.340+0000 7f5f5b09cb00 1 bucket sync caught up with source: 2021-09-17T10:13:47.320 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000003.31.12, , ] 2021-09-17T10:13:47.320 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000003.31.12, , ] 2021-09-17T10:13:47.321 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:47.340+0000 7f5f5b09cb00 0 bucket checkpoint complete 2021-09-17T10:13:47.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:47.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:47.352 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:47.402 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:47.402 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:47.450 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:47.465+0000 7f92a1f27b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:47.451 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:47.465+0000 7f92a1f27b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:47.451 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:47.465+0000 7f92a1f27b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:13:47.451 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:47.465+0000 7f92a1f27b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:47.503 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:47.517+0000 7f92a1f27b00 1 bucket sync caught up with source: 2021-09-17T10:13:47.503 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000003.31.12, , ] 2021-09-17T10:13:47.503 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000003.31.12, , ] 2021-09-17T10:13:47.504 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:47.517+0000 7f92a1f27b00 0 bucket checkpoint complete 2021-09-17T10:13:47.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:47.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:47.529 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:47.578 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:47.579 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:47.626 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:47.644+0000 7f1ca2de8b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:47.627 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:47.644+0000 7f1ca2de8b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:47.627 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:47.644+0000 7f1ca2de8b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:13:47.627 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:47.644+0000 7f1ca2de8b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:47.683 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:47.700+0000 7f1ca2de8b00 1 bucket sync caught up with source: 2021-09-17T10:13:47.683 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000003.31.12, , ] 2021-09-17T10:13:47.684 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000003.31.12, , ] 2021-09-17T10:13:47.684 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:47.700+0000 7f1ca2de8b00 0 bucket checkpoint complete 2021-09-17T10:13:47.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:47.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:47.715 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:47.765 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:47.766 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:47.812 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:47.825+0000 7f68aa94fb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:47.813 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:47.825+0000 7f68aa94fb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:47.813 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:47.825+0000 7f68aa94fb00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:13:47.813 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:47.825+0000 7f68aa94fb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:47.867 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:47.881+0000 7f68aa94fb00 1 bucket sync caught up with source: 2021-09-17T10:13:47.867 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000004.427.9, , ] 2021-09-17T10:13:47.868 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000004.427.9, , ] 2021-09-17T10:13:47.868 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:47.881+0000 7f68aa94fb00 0 bucket checkpoint complete 2021-09-17T10:13:47.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:47.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:47.896 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:47.945 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:47.946 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:47.993 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:48.005+0000 7eff32d43b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:47.993 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:48.005+0000 7eff32d43b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:47.993 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:48.005+0000 7eff32d43b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:13:47.993 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:48.005+0000 7eff32d43b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:48.049 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:48.061+0000 7eff32d43b00 1 bucket sync caught up with source: 2021-09-17T10:13:48.049 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000004.427.9, , ] 2021-09-17T10:13:48.049 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000004.427.9, , ] 2021-09-17T10:13:48.050 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:48.061+0000 7eff32d43b00 0 bucket checkpoint complete 2021-09-17T10:13:48.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:48.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:48.078 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:48.128 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:48.128 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:48.179 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:48.193+0000 7f35f2cfab00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:48.179 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:48.193+0000 7f35f2cfab00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:48.179 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:48.193+0000 7f35f2cfab00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:13:48.180 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:48.193+0000 7f35f2cfab00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:48.228 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:48.241+0000 7f35f2cfab00 1 bucket sync caught up with source: 2021-09-17T10:13:48.228 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000004.424.9, , ] 2021-09-17T10:13:48.229 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000004.424.9, , ] 2021-09-17T10:13:48.229 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:48.241+0000 7f35f2cfab00 0 bucket checkpoint complete 2021-09-17T10:13:48.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:48.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:48.256 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:48.307 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:48.307 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:48.359 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:48.376+0000 7fb4e0e0bb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:48.359 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:48.376+0000 7fb4e0e0bb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:48.359 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:48.376+0000 7fb4e0e0bb00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:13:48.359 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:48.376+0000 7fb4e0e0bb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:48.411 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:48.428+0000 7fb4e0e0bb00 1 bucket sync caught up with source: 2021-09-17T10:13:48.411 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000004.424.9, , ] 2021-09-17T10:13:48.411 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000004.424.9, , ] 2021-09-17T10:13:48.411 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:48.428+0000 7fb4e0e0bb00 0 bucket checkpoint complete 2021-09-17T10:13:48.441 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-20 zones={test-zone2, test-zone1} 2021-09-17T10:13:48.450 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:13:48.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:48.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:48.500 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:13:48.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:48.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:48.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:48.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:48.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:48.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:48.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:48.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:48.841 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-20 zones={test-zone2, test-zone1} 2021-09-17T10:13:48.842 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-20 zones={test-zone3, test-zone1} 2021-09-17T10:13:48.850 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:13:48.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:48.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:48.896 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:13:48.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:48.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:48.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:48.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:49.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:49.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:49.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:49.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:49.233 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-20 zones={test-zone3, test-zone1} 2021-09-17T10:13:49.234 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-20 zones={test-zone3, test-zone2} 2021-09-17T10:13:49.243 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:13:49.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:49.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:49.292 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:13:49.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:49.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:49.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:49.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:49.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:49.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:49.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:49.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:49.657 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-20 zones={test-zone3, test-zone2} 2021-09-17T10:13:49.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:49.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:49.672 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:49.715 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:49.715 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:49.797 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:13:49.829 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:13:49.830 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:49.830 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:49.831 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:13:49.831 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:49.831 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:49.831 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873629.687997_220.1", 2021-09-17T10:13:49.832 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:13:49.687997Z" 2021-09-17T10:13:49.832 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:49.832 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:49.832 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:49.832 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:49.833 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:49.833 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:49.833 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:49.833 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:49.834 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:49.834 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:13:49.835 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:13:49.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:49.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:49.837 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:49.890 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:49.891 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:49.983 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:49.983 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:49.983 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:49.984 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:49.984 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:49.984 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:49.984 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:49.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:49.985 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:49.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:49.985 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:49.985 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:49.986 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:49.986 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:49.986 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:49.986 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:13:49.987 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:49.987 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:49.987 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:49.987 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:49.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:49.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:49.988 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:49.988 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:49.988 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:49.989 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873629.687997_220.1", 2021-09-17T10:13:49.989 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:49.989 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:49.989 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:49.989 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:13:49.687997Z", 2021-09-17T10:13:49.990 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:49.990 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:49.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:49.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:49.991 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:49.991 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:49.991 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:49.991 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:49.991 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:49.992 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:49.992 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:49.992 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:49.992 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:49.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:49.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:49.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:49.993 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:49.993 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:49.994 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:49.994 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:49.994 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:49.994 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:49.995 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:49.995 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:49.995 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:49.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:49.996 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:49.996 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:49.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:49.996 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:49.996 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:13:49.997 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:13:49.997 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:49.997 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:49.998 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:13:49.999 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:13:49.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:50.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:50.000 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:50.043 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:50.044 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:50.155 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:13:50.156 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:13:50.156 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:13:50.156 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:13:50.156 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:13:50.156 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:50.157 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:50.157 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:50.157 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:13:50.157 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:50.158 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:13:50.158 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:50.158 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:50.158 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:50.159 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:50.159 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:13:50.159 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:50.159 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:50.160 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:50.160 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:50.160 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:50.160 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:50.161 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:13:50.161 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:50.161 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:50.161 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873629.687997_220.1", 2021-09-17T10:13:50.161 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:50.162 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:50.162 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:50.162 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:13:49.687997Z", 2021-09-17T10:13:50.162 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:50.163 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:50.163 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:50.163 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:50.164 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:13:50.164 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:50.164 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:50.164 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:50.164 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:50.165 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:50.165 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:50.165 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:13:50.165 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:50.166 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:50.166 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:50.166 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:50.166 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:13:50.167 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:50.167 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:50.167 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:50.167 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:50.168 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:50.168 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:50.168 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:13:50.168 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:50.168 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:50.169 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:50.169 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:13:50.169 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:50.169 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:13:50.170 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:13:50.170 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:13:50.170 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:50.170 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:13:50.172 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:13:50.172 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:13:50.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:50.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:50.174 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:50.217 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:50.217 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:50.332 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:50.332 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:50.333 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:50.334 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:50.335 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:13:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:50.336 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:50.337 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873629.687997_220.1", 2021-09-17T10:13:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:50.338 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:13:49.687997Z", 2021-09-17T10:13:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:50.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:50.340 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:50.341 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:50.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:50.343 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:50.344 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:50.345 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:50.346 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:13:50.347 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:13:50.347 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:50.347 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:50.348 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:13:50.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:50.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:50.400 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:50.450 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:50.450 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:50.492 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:50.512+0000 7f4e33184b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:50.492 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:50.512+0000 7f4e33184b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:50.492 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:50.512+0000 7f4e33184b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:13:50.492 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:50.512+0000 7f4e33184b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:50.542 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:50.560+0000 7f4e33184b00 1 bucket sync caught up with source: 2021-09-17T10:13:50.543 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000005.35.17, , ] 2021-09-17T10:13:50.543 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000005.35.17, , ] 2021-09-17T10:13:50.543 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:50.560+0000 7f4e33184b00 0 bucket checkpoint complete 2021-09-17T10:13:50.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:50.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:50.572 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:50.623 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:50.623 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:50.668 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:50.681+0000 7fcfeb87cb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:50.668 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:50.681+0000 7fcfeb87cb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:50.668 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:50.681+0000 7fcfeb87cb00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:13:50.669 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:50.681+0000 7fcfeb87cb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:50.714 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:50.729+0000 7fcfeb87cb00 1 bucket sync caught up with source: 2021-09-17T10:13:50.715 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000005.35.17, , ] 2021-09-17T10:13:50.715 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000005.35.17, , ] 2021-09-17T10:13:50.715 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:50.729+0000 7fcfeb87cb00 0 bucket checkpoint complete 2021-09-17T10:13:50.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:50.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:50.742 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:50.793 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:50.793 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:50.841 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:50.860+0000 7f9b79817b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:50.842 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:50.860+0000 7f9b79817b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:50.842 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:50.860+0000 7f9b79817b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:13:50.842 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:50.860+0000 7f9b79817b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:50.895 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:50.912+0000 7f9b79817b00 1 bucket sync caught up with source: 2021-09-17T10:13:50.895 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000005.35.17, , ] 2021-09-17T10:13:50.896 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000005.35.17, , ] 2021-09-17T10:13:50.896 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:50.912+0000 7f9b79817b00 0 bucket checkpoint complete 2021-09-17T10:13:50.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:50.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:50.925 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:50.974 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:50.975 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:51.022 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.037+0000 7faa4f414b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:51.023 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.037+0000 7faa4f414b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:51.023 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.037+0000 7faa4f414b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:13:51.023 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.037+0000 7faa4f414b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:51.079 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.093+0000 7faa4f414b00 1 bucket sync caught up with source: 2021-09-17T10:13:51.079 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000007.453.14, , ] 2021-09-17T10:13:51.080 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000007.453.14, , ] 2021-09-17T10:13:51.080 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.093+0000 7faa4f414b00 0 bucket checkpoint complete 2021-09-17T10:13:51.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:51.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:51.108 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:51.153 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:51.153 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:51.201 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.213+0000 7f43599e6b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:51.201 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.213+0000 7f43599e6b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:51.201 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.213+0000 7f43599e6b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:13:51.202 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.213+0000 7f43599e6b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:51.255 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.269+0000 7f43599e6b00 1 bucket sync caught up with source: 2021-09-17T10:13:51.255 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000007.453.14, , ] 2021-09-17T10:13:51.256 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000007.453.14, , ] 2021-09-17T10:13:51.256 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.269+0000 7f43599e6b00 0 bucket checkpoint complete 2021-09-17T10:13:51.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:51.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:51.287 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:51.334 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:51.334 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:51.382 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.397+0000 7f34aa6cab00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:51.382 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.397+0000 7f34aa6cab00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:51.382 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.397+0000 7f34aa6cab00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:13:51.382 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.397+0000 7f34aa6cab00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:51.436 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.449+0000 7f34aa6cab00 1 bucket sync caught up with source: 2021-09-17T10:13:51.437 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000007.450.14, , ] 2021-09-17T10:13:51.437 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000007.450.14, , ] 2021-09-17T10:13:51.437 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:51.449+0000 7f34aa6cab00 0 bucket checkpoint complete 2021-09-17T10:13:51.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:51.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:51.465 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:51.516 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:51.517 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:51.568 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:51.587+0000 7fd48016db00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:51.569 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:51.587+0000 7fd48016db00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:51.569 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:51.587+0000 7fd48016db00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:13:51.569 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:51.587+0000 7fd48016db00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:51.643+0000 7fd48016db00 1 bucket sync caught up with source: 2021-09-17T10:13:51.627 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000007.450.14, , ] 2021-09-17T10:13:51.627 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000007.450.14, , ] 2021-09-17T10:13:51.627 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:51.643+0000 7fd48016db00 0 bucket checkpoint complete 2021-09-17T10:13:51.655 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-20 zones={test-zone2, test-zone1} 2021-09-17T10:13:51.664 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:13:51.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:51.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:51.712 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:13:51.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:51.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:51.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:51.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:51.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:51.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:51.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:51.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.052 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-20 zones={test-zone2, test-zone1} 2021-09-17T10:13:52.053 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-20 zones={test-zone3, test-zone1} 2021-09-17T10:13:52.061 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:13:52.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:52.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:52.112 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:13:52.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:52.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:52.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.445 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-20 zones={test-zone3, test-zone1} 2021-09-17T10:13:52.446 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-20 zones={test-zone3, test-zone2} 2021-09-17T10:13:52.456 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:13:52.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:52.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:52.504 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:13:52.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:52.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:13:52.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:13:52.849 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-20 zones={test-zone3, test-zone2} 2021-09-17T10:13:52.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-09-17T10:13:52.851 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-21 2021-09-17T10:13:52.879 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-22 2021-09-17T10:13:52.932 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-23 2021-09-17T10:13:53.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:53.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:53.017 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:53.051 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:53.051 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:53.136 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:13:53.167 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:13:53.168 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:53.168 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:53.168 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:13:53.169 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:53.169 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:53.169 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873633.032261_277.1", 2021-09-17T10:13:53.169 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:13:53.032261Z" 2021-09-17T10:13:53.170 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:53.170 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:53.170 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:53.170 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:53.170 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:53.171 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:53.171 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:53.171 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:13:53.171 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:53.172 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:13:53.173 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:13:53.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:53.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:53.174 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:53.233 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:53.234 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:53.455 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:53.455 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:53.456 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:53.456 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:53.456 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:53.456 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:53.457 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:53.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:53.457 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:53.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:53.458 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:53.458 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:53.458 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:53.458 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:13:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:53.459 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:53.460 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873632.918166_264.1", 2021-09-17T10:13:53.461 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:13:52.918166Z", 2021-09-17T10:13:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:53.462 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:53.463 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:53.464 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:53.465 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:53.465 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:53.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:53.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:53.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:53.466 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:53.466 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:53.466 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:53.466 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:53.467 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:53.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:53.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:53.468 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:53.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:53.469 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:53.469 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:13:53.469 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:13:53.469 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:53.469 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:53.470 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631873633.032261_277.1 target=1_1631873632.918166_264.1 2021-09-17T10:13:58.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:58.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:58.474 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:58.513 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:58.513 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:58.632 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:58.632 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:58.633 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:58.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:58.635 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:13:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:58.636 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:58.637 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:58.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:58.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:58.637 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:58.638 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:58.638 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:58.638 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873633.032261_277.1", 2021-09-17T10:13:58.638 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:58.639 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:58.639 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:58.639 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:13:53.032261Z", 2021-09-17T10:13:58.639 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:58.640 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:58.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:58.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:58.640 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:58.641 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:58.641 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:58.641 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:58.641 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:58.641 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:58.642 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:58.642 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:58.642 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:58.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:58.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:58.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:58.643 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:58.643 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:58.644 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:58.644 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:58.644 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:58.644 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:58.645 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:58.645 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:58.645 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:58.645 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:58.646 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:58.646 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:58.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:58.646 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:58.646 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:13:58.647 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:13:58.647 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:58.647 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:58.648 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:13:58.649 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:13:58.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:58.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:58.651 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:58.692 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:58.693 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:58.806 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:13:58.807 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:13:58.807 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:13:58.807 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:13:58.807 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:13:58.808 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:58.808 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:58.808 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:58.808 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:13:58.809 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:58.809 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:13:58.809 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:58.809 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:58.810 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:58.810 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:58.810 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:13:58.810 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:58.811 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:58.811 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:58.811 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:58.811 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:58.811 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:58.812 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:13:58.812 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:58.812 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:58.812 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873633.032261_277.1", 2021-09-17T10:13:58.812 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:58.813 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:58.813 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:58.813 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:13:53.032261Z", 2021-09-17T10:13:58.813 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:58.814 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:58.814 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:58.814 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:58.814 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:13:58.815 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:58.815 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:58.815 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:58.815 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:58.815 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:58.816 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:58.816 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:13:58.816 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:58.816 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:58.817 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:58.817 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:13:58.817 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:13:58.817 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:13:58.817 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:13:58.818 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:13:58.818 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:13:58.818 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:13:58.818 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:13:58.819 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:13:58.819 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:13:58.819 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:58.819 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:58.819 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:13:58.820 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:13:58.820 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:13:58.820 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:13:58.820 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:13:58.821 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:13:58.821 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:13:58.822 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:13:58.822 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:13:58.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:58.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:13:58.824 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:13:58.867 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:58.868 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:58.984 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:13:58.985 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:13:58.985 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:13:58.985 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:13:58.986 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:13:58.986 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:13:58.986 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:58.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:58.987 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:13:58.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:58.987 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:13:58.988 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:58.988 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:58.988 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:13:58.989 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:58.989 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:13:58.989 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:58.989 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:13:58.990 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:58.990 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:58.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:58.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:13:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873633.032261_277.1", 2021-09-17T10:13:58.991 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:13:53.032261Z", 2021-09-17T10:13:58.992 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:58.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:58.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:58.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:58.993 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:13:58.993 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:58.994 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:58.994 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:58.994 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:58.995 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:58.995 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:58.995 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:58.995 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:58.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:58.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:58.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:13:58.996 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:13:58.996 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:13:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:13:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:13:58.997 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:13:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:13:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:13:58.998 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:13:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:13:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:58.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:13:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:13:59.000 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:13:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:13:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:13:59.001 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:13:59.001 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:13:59.003 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:13:59.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:59.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:59.208 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:59.257 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:59.257 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:59.297 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:59.315+0000 7f6202395b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:59.297 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:59.315+0000 7f6202395b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:59.297 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:59.315+0000 7f6202395b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:13:59.298 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:59.315+0000 7f6202395b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:59.337 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:59.355+0000 7f6202395b00 1 bucket sync caught up with source: 2021-09-17T10:13:59.338 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000003.14.11, , ] 2021-09-17T10:13:59.338 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000003.14.11, , ] 2021-09-17T10:13:59.338 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:59.355+0000 7f6202395b00 0 bucket checkpoint complete 2021-09-17T10:13:59.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:59.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:59.365 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:59.418 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:59.418 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:59.461 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:59.481+0000 7fec12f69b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:59.462 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:59.481+0000 7fec12f69b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:59.462 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:59.481+0000 7fec12f69b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:13:59.462 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:59.481+0000 7fec12f69b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:59.507 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:59.525+0000 7fec12f69b00 1 bucket sync caught up with source: 2021-09-17T10:13:59.508 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000003.14.11, , ] 2021-09-17T10:13:59.508 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000003.14.11, , ] 2021-09-17T10:13:59.508 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:59.525+0000 7fec12f69b00 0 bucket checkpoint complete 2021-09-17T10:13:59.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:59.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:59.530 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:59.581 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:59.634 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:59.651+0000 7f50ea6f4b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:59.634 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:59.651+0000 7f50ea6f4b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:59.634 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:59.651+0000 7f50ea6f4b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:13:59.635 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:59.651+0000 7f50ea6f4b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:59.688 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:59.707+0000 7f50ea6f4b00 1 bucket sync caught up with source: 2021-09-17T10:13:59.754 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000003.14.11, , ] 2021-09-17T10:13:59.754 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000003.14.11, , ] 2021-09-17T10:13:59.755 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:13:59.707+0000 7f50ea6f4b00 0 bucket checkpoint complete 2021-09-17T10:13:59.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:59.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:59.757 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:59.806 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:59.807 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:13:59.855 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:59.873+0000 7fc267332b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:13:59.855 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:59.873+0000 7fc267332b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:13:59.856 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:59.873+0000 7fc267332b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:13:59.856 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:59.873+0000 7fc267332b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:13:59.908 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:59.925+0000 7fc267332b00 1 bucket sync caught up with source: 2021-09-17T10:13:59.908 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000002.187.5, , ] 2021-09-17T10:13:59.909 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000002.187.5, , ] 2021-09-17T10:13:59.909 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:13:59.925+0000 7fc267332b00 0 bucket checkpoint complete 2021-09-17T10:13:59.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:59.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:13:59.939 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:13:59.989 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:13:59.989 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:00.035 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:00.053+0000 7f4429862b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:00.036 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:00.053+0000 7f4429862b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:00.036 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:00.053+0000 7f4429862b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:14:00.036 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:00.053+0000 7f4429862b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:00.076 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:00.093+0000 7f4429862b00 1 bucket sync caught up with source: 2021-09-17T10:14:00.076 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000002.187.5, , ] 2021-09-17T10:14:00.076 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000002.187.5, , ] 2021-09-17T10:14:00.077 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:00.093+0000 7f4429862b00 0 bucket checkpoint complete 2021-09-17T10:14:00.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:00.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:00.101 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:00.148 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:00.148 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:00.191 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:00.209+0000 7f717abfbb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:00.191 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:00.209+0000 7f717abfbb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:00.191 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:00.209+0000 7f717abfbb00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:14:00.192 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:00.209+0000 7f717abfbb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:00.242 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:00.261+0000 7f717abfbb00 1 bucket sync caught up with source: 2021-09-17T10:14:00.242 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000002.265.5, , ] 2021-09-17T10:14:00.243 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000002.265.5, , ] 2021-09-17T10:14:00.243 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:00.261+0000 7f717abfbb00 0 bucket checkpoint complete 2021-09-17T10:14:00.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:00.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:00.266 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:00.318 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:00.318 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:00.359 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:00.375+0000 7f685fc82b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:00.360 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:00.375+0000 7f685fc82b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:00.360 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:00.375+0000 7f685fc82b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:14:00.360 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:00.375+0000 7f685fc82b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:00.414 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:00.431+0000 7f685fc82b00 1 bucket sync caught up with source: 2021-09-17T10:14:00.415 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000002.265.5, , ] 2021-09-17T10:14:00.415 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000002.265.5, , ] 2021-09-17T10:14:00.415 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:00.431+0000 7f685fc82b00 0 bucket checkpoint complete 2021-09-17T10:14:00.445 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-21 zones={test-zone2, test-zone1} 2021-09-17T10:14:00.455 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:14:00.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:00.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:00.508 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:14:00.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:00.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:00.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:00.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:00.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:00.666 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-21 zones={test-zone2, test-zone1} 2021-09-17T10:14:00.667 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-21 zones={test-zone3, test-zone1} 2021-09-17T10:14:00.674 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:14:00.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:00.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:00.720 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:14:00.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:00.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:00.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:00.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:00.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:00.871 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-21 zones={test-zone3, test-zone1} 2021-09-17T10:14:00.872 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-21 zones={test-zone3, test-zone2} 2021-09-17T10:14:00.880 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:14:00.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:00.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:00.933 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:14:00.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:00.942 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:00.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:01.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:01.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:01.090 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-21 zones={test-zone3, test-zone2} 2021-09-17T10:14:01.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:01.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:01.092 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:01.141 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:01.141 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:01.192 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:01.207+0000 7f9ecd20fb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:01.192 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:01.207+0000 7f9ecd20fb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:01.192 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:01.207+0000 7f9ecd20fb00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:14:01.192 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:01.207+0000 7f9ecd20fb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:01.247 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:01.263+0000 7f9ecd20fb00 1 bucket sync caught up with source: 2021-09-17T10:14:01.248 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000002.35.5, , ] 2021-09-17T10:14:01.248 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000002.35.5, , ] 2021-09-17T10:14:01.248 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:01.263+0000 7f9ecd20fb00 0 bucket checkpoint complete 2021-09-17T10:14:01.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:01.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:01.282 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:01.331 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:01.332 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:01.385 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.401+0000 7fbcf2043b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:01.385 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.401+0000 7fbcf2043b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:01.386 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.401+0000 7fbcf2043b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:14:01.386 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.401+0000 7fbcf2043b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:01.442 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.461+0000 7fbcf2043b00 1 bucket sync caught up with source: 2021-09-17T10:14:01.442 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000002.35.5, , ] 2021-09-17T10:14:01.442 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000002.35.5, , ] 2021-09-17T10:14:01.443 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.461+0000 7fbcf2043b00 0 bucket checkpoint complete 2021-09-17T10:14:01.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:01.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:01.470 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:01.587+0000 7f1f2bd86b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:01.568 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:01.587+0000 7f1f2bd86b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:01.569 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:01.587+0000 7f1f2bd86b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:14:01.569 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:01.587+0000 7f1f2bd86b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:01.624 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:01.639+0000 7f1f2bd86b00 1 bucket sync caught up with source: 2021-09-17T10:14:01.624 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000002.35.5, , ] 2021-09-17T10:14:01.625 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000002.35.5, , ] 2021-09-17T10:14:01.625 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:01.639+0000 7f1f2bd86b00 0 bucket checkpoint complete 2021-09-17T10:14:01.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:01.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:01.656 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:01.708 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:01.708 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:01.760 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.777+0000 7f987726db00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:01.760 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.777+0000 7f987726db00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:01.760 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.777+0000 7f987726db00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:14:01.761 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.777+0000 7f987726db00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:01.816 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.833+0000 7f987726db00 1 bucket sync caught up with source: 2021-09-17T10:14:01.817 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000003.168.11, , ] 2021-09-17T10:14:01.817 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000003.168.11, , ] 2021-09-17T10:14:01.817 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.833+0000 7f987726db00 0 bucket checkpoint complete 2021-09-17T10:14:01.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:01.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:01.849 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:01.899 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:01.900 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:01.948 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.965+0000 7ff713e02b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:01.948 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.965+0000 7ff713e02b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:01.948 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.965+0000 7ff713e02b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:14:01.949 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:01.965+0000 7ff713e02b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:02.003 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:02.021+0000 7ff713e02b00 1 bucket sync caught up with source: 2021-09-17T10:14:02.004 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000003.168.11, , ] 2021-09-17T10:14:02.004 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000003.168.11, , ] 2021-09-17T10:14:02.004 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:02.021+0000 7ff713e02b00 0 bucket checkpoint complete 2021-09-17T10:14:02.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:02.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:02.037 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:02.089 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:02.089 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:02.137 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:02.157+0000 7f8e10a53b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:02.137 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:02.157+0000 7f8e10a53b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:02.138 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:02.157+0000 7f8e10a53b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:14:02.138 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:02.157+0000 7f8e10a53b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:02.191 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:02.209+0000 7f8e10a53b00 1 bucket sync caught up with source: 2021-09-17T10:14:02.191 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000002.181.5, , ] 2021-09-17T10:14:02.191 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000002.181.5, , ] 2021-09-17T10:14:02.192 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:02.209+0000 7f8e10a53b00 0 bucket checkpoint complete 2021-09-17T10:14:02.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:02.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:02.216 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:02.266 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:02.266 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:02.313 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:02.331+0000 7f662b89bb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:02.314 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:02.331+0000 7f662b89bb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:02.314 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:02.331+0000 7f662b89bb00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:14:02.315 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:02.331+0000 7f662b89bb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:02.383+0000 7f662b89bb00 1 bucket sync caught up with source: 2021-09-17T10:14:02.366 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000002.181.5, , ] 2021-09-17T10:14:02.366 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000002.181.5, , ] 2021-09-17T10:14:02.366 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:02.383+0000 7f662b89bb00 0 bucket checkpoint complete 2021-09-17T10:14:02.395 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-22 zones={test-zone2, test-zone1} 2021-09-17T10:14:02.403 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:14:02.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:02.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:02.452 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:14:02.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:02.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:02.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:02.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:02.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:02.600 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-22 zones={test-zone2, test-zone1} 2021-09-17T10:14:02.601 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-22 zones={test-zone3, test-zone1} 2021-09-17T10:14:02.609 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:14:02.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:02.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:02.656 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:14:02.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:02.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:02.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:02.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:02.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:02.801 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-22 zones={test-zone3, test-zone1} 2021-09-17T10:14:02.802 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-22 zones={test-zone3, test-zone2} 2021-09-17T10:14:02.810 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:14:02.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:02.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:02.861 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:14:02.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:02.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:02.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:02.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:02.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:03.013 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-22 zones={test-zone3, test-zone2} 2021-09-17T10:14:03.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:03.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:03.015 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:03.062 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:03.062 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:03.126+0000 7f064e677b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:03.126+0000 7f064e677b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:03.126+0000 7f064e677b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:14:03.111 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:03.126+0000 7f064e677b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:03.166 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:03.182+0000 7f064e677b00 1 bucket sync caught up with source: 2021-09-17T10:14:03.166 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000002.35.5, , ] 2021-09-17T10:14:03.167 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000002.35.5, , ] 2021-09-17T10:14:03.167 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:03.182+0000 7f064e677b00 0 bucket checkpoint complete 2021-09-17T10:14:03.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:03.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:03.198 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:03.247 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:03.248 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:03.291 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.309+0000 7fb5c9230b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:03.291 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.309+0000 7fb5c9230b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:03.291 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.309+0000 7fb5c9230b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:14:03.292 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.309+0000 7fb5c9230b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:03.344 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.361+0000 7fb5c9230b00 1 bucket sync caught up with source: 2021-09-17T10:14:03.344 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000002.35.5, , ] 2021-09-17T10:14:03.344 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000002.35.5, , ] 2021-09-17T10:14:03.345 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.361+0000 7fb5c9230b00 0 bucket checkpoint complete 2021-09-17T10:14:03.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:03.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:03.372 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:03.423 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:03.469 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:03.486+0000 7f1899fe7b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:03.470 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:03.486+0000 7f1899fe7b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:03.470 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:03.486+0000 7f1899fe7b00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:14:03.470 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:03.486+0000 7f1899fe7b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:03.526 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:03.542+0000 7f1899fe7b00 1 bucket sync caught up with source: 2021-09-17T10:14:03.526 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000002.35.5, , ] 2021-09-17T10:14:03.527 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000002.35.5, , ] 2021-09-17T10:14:03.527 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:03.542+0000 7f1899fe7b00 0 bucket checkpoint complete 2021-09-17T10:14:03.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:03.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:03.557 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:03.607 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:03.607 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:03.657 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.677+0000 7f2624133b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:03.658 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.677+0000 7f2624133b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:03.658 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.677+0000 7f2624133b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:14:03.658 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.677+0000 7f2624133b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:03.714 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.733+0000 7f2624133b00 1 bucket sync caught up with source: 2021-09-17T10:14:03.714 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000002.232.5, , ] 2021-09-17T10:14:03.714 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000002.232.5, , ] 2021-09-17T10:14:03.715 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.733+0000 7f2624133b00 0 bucket checkpoint complete 2021-09-17T10:14:03.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:03.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:03.744 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:03.793 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:03.794 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:03.840 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.857+0000 7f8c8c48fb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:03.840 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.857+0000 7f8c8c48fb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:03.840 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.857+0000 7f8c8c48fb00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:14:03.840 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.857+0000 7f8c8c48fb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:03.894 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.913+0000 7f8c8c48fb00 1 bucket sync caught up with source: 2021-09-17T10:14:03.894 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000002.232.5, , ] 2021-09-17T10:14:03.894 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000002.232.5, , ] 2021-09-17T10:14:03.895 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:03.913+0000 7f8c8c48fb00 0 bucket checkpoint complete 2021-09-17T10:14:03.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:03.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:03.926 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:03.975 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:03.976 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:04.026 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:04.045+0000 7f231ce2db00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:04.027 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:04.045+0000 7f231ce2db00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:04.027 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:04.045+0000 7f231ce2db00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:14:04.027 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:04.045+0000 7f231ce2db00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:04.081 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:04.101+0000 7f231ce2db00 1 bucket sync caught up with source: 2021-09-17T10:14:04.081 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000003.214.11, , ] 2021-09-17T10:14:04.082 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000003.214.11, , ] 2021-09-17T10:14:04.082 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:04.101+0000 7f231ce2db00 0 bucket checkpoint complete 2021-09-17T10:14:04.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:04.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:04.107 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:04.157 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:04.157 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:04.200 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:04.214+0000 7ff1e70b4b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:04.200 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:04.214+0000 7ff1e70b4b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:04.200 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:04.214+0000 7ff1e70b4b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:14:04.201 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:04.214+0000 7ff1e70b4b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:04.255 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:04.270+0000 7ff1e70b4b00 1 bucket sync caught up with source: 2021-09-17T10:14:04.255 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000003.214.11, , ] 2021-09-17T10:14:04.255 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000003.214.11, , ] 2021-09-17T10:14:04.256 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:04.270+0000 7ff1e70b4b00 0 bucket checkpoint complete 2021-09-17T10:14:04.283 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-23 zones={test-zone2, test-zone1} 2021-09-17T10:14:04.293 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:14:04.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:04.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:04.340 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:14:04.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:04.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:04.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:04.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:04.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:04.488 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-23 zones={test-zone2, test-zone1} 2021-09-17T10:14:04.489 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-23 zones={test-zone3, test-zone1} 2021-09-17T10:14:04.495 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:14:04.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:04.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:04.544 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:14:04.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:04.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:04.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:04.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:04.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:04.690 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-23 zones={test-zone3, test-zone1} 2021-09-17T10:14:04.690 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-23 zones={test-zone3, test-zone2} 2021-09-17T10:14:04.699 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:14:04.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:04.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:04.748 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:14:04.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:04.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:04.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:04.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:04.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:04.893 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gopfzo-23 zones={test-zone3, test-zone2} 2021-09-17T10:14:04.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-09-17T10:14:04.894 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-24 2021-09-17T10:14:04.922 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-25 2021-09-17T10:14:04.971 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-26 2021-09-17T10:14:05.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:05.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:05.107 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:05.157 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:05.158 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:05.354 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:14:05.415 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:14:05.415 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:05.416 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:05.416 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:14:05.416 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:05.416 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:05.417 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873645.117756_304.1", 2021-09-17T10:14:05.417 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:14:05.117756Z" 2021-09-17T10:14:05.417 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:05.417 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:05.417 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:05.418 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:14:05.418 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:05.418 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:05.418 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:05.419 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:14:05.419 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:05.419 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:14:05.420 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:14:05.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:05.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:05.422 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:05.520 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:05.520 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:05.733 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:14:05.734 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:14:05.734 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:14:05.734 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:14:05.735 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:14:05.735 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:05.735 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:05.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:05.735 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:14:05.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:05.736 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:14:05.736 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:05.736 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:05.737 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:05.737 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:05.737 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:14:05.737 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:05.737 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:05.738 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:05.738 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:05.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:05.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:05.739 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:14:05.739 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:05.739 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:05.739 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873645.117756_304.1", 2021-09-17T10:14:05.740 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:05.740 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:05.740 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:05.740 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:14:05.117756Z", 2021-09-17T10:14:05.740 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:05.741 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:05.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:05.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:05.741 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:14:05.742 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:05.742 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:05.742 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:14:05.742 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:05.742 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:05.743 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:05.743 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:14:05.743 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:05.743 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:05.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:05.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:05.744 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:14:05.744 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:05.744 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:05.745 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:14:05.745 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:05.745 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:05.745 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:05.746 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:14:05.746 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:05.746 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:05.746 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:05.746 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:14:05.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:05.747 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:14:05.747 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:14:05.747 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:14:05.748 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:05.748 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:14:05.749 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:14:05.750 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:14:05.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:05.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:05.751 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:05.795 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:05.795 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:05.909 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:14:05.910 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:14:05.910 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:14:05.910 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:14:05.910 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:14:05.911 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:05.911 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:05.911 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:05.911 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:14:05.912 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:05.912 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:14:05.912 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:05.912 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:05.913 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:05.913 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:05.913 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:14:05.913 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:05.914 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:05.914 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:05.914 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:05.914 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:05.915 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:05.915 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:14:05.915 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:05.915 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:05.916 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873644.961263_285.1", 2021-09-17T10:14:05.916 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:05.916 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:05.916 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:05.917 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:04.961263Z", 2021-09-17T10:14:05.917 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:05.917 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:05.917 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:05.918 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:05.918 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:14:05.918 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:05.918 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:05.919 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:05.919 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:05.919 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:05.919 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:05.919 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:05.920 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:05.920 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:05.920 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:05.920 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:05.921 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:14:05.921 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:05.921 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:05.921 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:05.922 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:05.922 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:05.922 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:05.922 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:05.923 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:05.923 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:05.923 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:05.923 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:14:05.924 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:05.924 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:14:05.924 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:14:05.924 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:14:05.925 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:05.925 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:14:05.926 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873645.117756_304.1 target=1_1631873644.961263_285.1 2021-09-17T10:14:10.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:10.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:10.929 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:10.972 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:10.972 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:11.090 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:14:11.091 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:14:11.091 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:14:11.091 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:14:11.091 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:14:11.092 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:11.092 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:11.092 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:11.092 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:14:11.092 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:11.093 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:14:11.093 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:11.093 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:11.093 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:11.094 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:11.094 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:14:11.095 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:11.095 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:11.095 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:11.095 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:11.096 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:11.096 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:11.096 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:14:11.096 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:11.096 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:11.097 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873645.117756_304.1", 2021-09-17T10:14:11.097 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:11.097 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:11.097 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:11.098 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:05.117756Z", 2021-09-17T10:14:11.098 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:11.098 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:11.098 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:11.098 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:11.099 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:14:11.099 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:11.099 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:11.099 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:11.100 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:11.100 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:11.100 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:11.100 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:11.101 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:11.101 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:11.101 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:11.101 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:11.101 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:14:11.102 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:11.102 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:11.102 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:11.102 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:11.103 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:11.103 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:11.103 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:11.103 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:11.103 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:11.104 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:11.104 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:14:11.104 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:11.104 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:14:11.105 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:14:11.105 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:14:11.105 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:11.105 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:14:11.107 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-17T10:14:11.107 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-17T10:14:11.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:11.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:11.109 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:11.151 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:11.152 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:11.270 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:14:11.270 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:14:11.270 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:14:11.271 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:14:11.271 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:14:11.271 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:11.271 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:11.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:11.272 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:14:11.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:11.272 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:14:11.273 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:11.273 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:11.273 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:11.273 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:11.274 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:14:11.274 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:11.274 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:11.274 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:11.275 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:11.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:11.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:11.275 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:14:11.276 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:11.276 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:11.276 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873645.117756_304.1", 2021-09-17T10:14:11.276 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:11.277 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:11.277 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:11.277 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:14:05.117756Z", 2021-09-17T10:14:11.277 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:11.278 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:11.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:11.278 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:11.278 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:14:11.279 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:11.279 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:11.279 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:14:11.279 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:11.279 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:11.280 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:11.280 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:14:11.280 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:11.280 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:11.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:11.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:11.281 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:14:11.281 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:11.282 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:11.282 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:14:11.282 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:11.282 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:11.283 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:11.283 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:14:11.283 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:11.283 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:11.284 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:11.284 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:14:11.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:11.284 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:14:11.285 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:14:11.285 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:14:11.285 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:11.285 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:14:11.287 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-17T10:14:11.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:11.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:11.503 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:11.551 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:11.551 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:11.589 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:11.606+0000 7f0f7030eb00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:11.589 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:11.606+0000 7f0f7030eb00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:11.589 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:11.606+0000 7f0f7030eb00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:14:11.589 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:11.606+0000 7f0f7030eb00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:11.632 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:11.646+0000 7f0f7030eb00 1 bucket sync caught up with source: 2021-09-17T10:14:11.633 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000003.29.6, , ] 2021-09-17T10:14:11.633 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000003.29.6, , ] 2021-09-17T10:14:11.633 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:11.646+0000 7f0f7030eb00 0 bucket checkpoint complete 2021-09-17T10:14:11.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:11.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:11.662 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:11.715 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:11.715 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:11.761 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:11.780+0000 7f1946ed3b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:11.762 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:11.780+0000 7f1946ed3b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:11.762 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:11.780+0000 7f1946ed3b00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:14:11.762 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:11.780+0000 7f1946ed3b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:11.814 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:11.832+0000 7f1946ed3b00 1 bucket sync caught up with source: 2021-09-17T10:14:11.815 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000003.29.6, , ] 2021-09-17T10:14:11.815 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000003.29.6, , ] 2021-09-17T10:14:11.815 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:11.832+0000 7f1946ed3b00 0 bucket checkpoint complete 2021-09-17T10:14:11.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:11.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:11.843 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:11.891 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:11.892 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:11.941 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:11.958+0000 7fecfebeab00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:11.941 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:11.958+0000 7fecfebeab00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:11.941 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:11.958+0000 7fecfebeab00 1 Zone: test-zone4 (74241c6a-1297-4f4a-8bbd-b65c661636d9) 2021-09-17T10:14:11.942 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:11.958+0000 7fecfebeab00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:11.997 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:12.014+0000 7fecfebeab00 1 bucket sync caught up with source: 2021-09-17T10:14:11.998 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000003.29.6, , ] 2021-09-17T10:14:11.998 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000003.29.6, , ] 2021-09-17T10:14:11.998 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:12.014+0000 7fecfebeab00 0 bucket checkpoint complete 2021-09-17T10:14:12.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:12.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:12.028 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:12.079 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:12.079 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:12.121 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.136+0000 7f4666830b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:12.121 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.136+0000 7f4666830b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:12.121 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.136+0000 7f4666830b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:14:12.122 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.136+0000 7f4666830b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:12.172 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.188+0000 7f4666830b00 1 bucket sync caught up with source: 2021-09-17T10:14:12.172 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000001.81.10, , ] 2021-09-17T10:14:12.173 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000001.81.10, , ] 2021-09-17T10:14:12.173 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.188+0000 7f4666830b00 0 bucket checkpoint complete 2021-09-17T10:14:12.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:12.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:12.206 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:12.253 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:12.254 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:12.295 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.312+0000 7f700f74ab00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:12.295 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.312+0000 7f700f74ab00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:12.296 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.312+0000 7f700f74ab00 1 Zone: test-zone3 (d9facd38-bb4e-4ea2-9817-2655cc5c6379) 2021-09-17T10:14:12.296 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.312+0000 7f700f74ab00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:12.348 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.364+0000 7f700f74ab00 1 bucket sync caught up with source: 2021-09-17T10:14:12.349 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000001.81.10, , ] 2021-09-17T10:14:12.349 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000001.81.10, , ] 2021-09-17T10:14:12.349 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.364+0000 7f700f74ab00 0 bucket checkpoint complete 2021-09-17T10:14:12.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:12.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:12.382 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket gopfzo-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:12.433 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:12.433 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:12.482 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.500+0000 7f537bfa5b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:12.482 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.500+0000 7f537bfa5b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:12.483 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.500+0000 7f537bfa5b00 1 Zone: test-zone1 (9854c5a3-6cc0-45a7-b530-3af2b1c733d5) 2021-09-17T10:14:12.483 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.500+0000 7f537bfa5b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:12.537 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.556+0000 7f537bfa5b00 1 bucket sync caught up with source: 2021-09-17T10:14:12.538 INFO:teuthology.orchestra.run.smithi018.stderr: local status: [, , , , , , , , 00000000002.427.12, , ] 2021-09-17T10:14:12.538 INFO:teuthology.orchestra.run.smithi018.stderr: remote markers: [, , , , , , , , 00000000002.427.12, , ] 2021-09-17T10:14:12.538 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:14:12.556+0000 7f537bfa5b00 0 bucket checkpoint complete 2021-09-17T10:14:12.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:12.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'gopfzo-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-17T10:14:12.567 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket gopfzo-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-17T10:14:12.615 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:12.616 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:12.665 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:12.682+0000 7f7d1e615b00 1 Realm: test-realm (3c1ea6e8-af17-477a-9c20-483e62be612c) 2021-09-17T10:14:12.666 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:12.682+0000 7f7d1e615b00 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-17T10:14:12.666 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:12.682+0000 7f7d1e615b00 1 Zone: test-zone2 (304080b6-3b8b-4555-8935-e35e25942a93) 2021-09-17T10:14:12.666 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:12.682+0000 7f7d1e615b00 1 using period configuration: 3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac:4 2021-09-17T10:14:12.721 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:12.738+0000 7f7d1e615b00 1 bucket sync caught up with source: 2021-09-17T10:14:12.721 INFO:teuthology.orchestra.run.smithi050.stderr: local status: [, , , , , , , , 00000000002.427.12, , ] 2021-09-17T10:14:12.722 INFO:teuthology.orchestra.run.smithi050.stderr: remote markers: [, , , , , , , , 00000000002.427.12, , ] 2021-09-17T10:14:12.722 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:14:12.738+0000 7f7d1e615b00 0 bucket checkpoint complete 2021-09-17T10:14:12.753 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gopfzo-24 zones={test-zone2, test-zone1} 2021-09-17T10:14:12.762 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-17T10:14:12.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:12.812 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-17T10:14:12.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-17T10:14:12.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-17T10:14:12.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... FAIL 2021-09-17T10:14:12.875 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-27 2021-09-17T10:14:12.907 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-28 2021-09-17T10:14:12.960 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-29 2021-09-17T10:14:13.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2021-09-17T10:14:13.199 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-30 2021-09-17T10:14:13.254 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-31 2021-09-17T10:14:13.368 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-32 2021-09-17T10:14:13.611 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2021-09-17T10:14:13.612 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-33 2021-09-17T10:14:13.689 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-34 2021-09-17T10:14:14.104 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-35 2021-09-17T10:14:16.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2021-09-17T10:14:16.428 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-36 2021-09-17T10:14:16.533 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-37 2021-09-17T10:14:16.668 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-38 2021-09-17T10:14:16.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:16.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:16.842 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:16.886 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:16.887 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:16.948 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:14:16.969 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:14:16.969 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:16.970 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:16.970 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:14:16.970 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:16.970 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:16.971 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873656.806856_380.1", 2021-09-17T10:14:16.971 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:14:16.806856Z" 2021-09-17T10:14:16.971 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:16.971 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:16.972 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:16.972 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:14:16.972 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:16.972 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:16.972 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:16.973 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:14:16.973 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:16.973 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:14:16.974 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:14:16.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:16.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:16.976 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:17.030 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:17.031 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:17.161 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:14:17.162 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:14:17.162 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:14:17.162 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:14:17.162 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:14:17.163 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:17.163 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:17.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:17.163 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:14:17.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:17.164 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:14:17.164 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:17.164 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:17.164 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:17.165 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:17.165 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:14:17.165 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:17.165 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:17.166 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:17.166 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:17.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:17.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:17.166 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:14:17.167 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:17.167 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:17.167 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873656.597705_373.1", 2021-09-17T10:14:17.167 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:17.168 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:17.168 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:17.168 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:14:16.597705Z", 2021-09-17T10:14:17.168 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:17.169 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:17.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:17.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:17.169 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:14:17.169 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:17.170 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:17.170 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:14:17.170 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:17.170 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:17.171 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:17.171 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:14:17.171 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:17.171 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:17.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:17.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:17.172 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:14:17.172 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:17.172 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:17.172 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:14:17.173 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:17.173 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:17.173 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:17.173 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:14:17.174 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:17.174 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:17.174 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:17.174 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:14:17.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:17.175 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:14:17.175 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:14:17.175 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:14:17.175 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:17.175 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:14:17.178 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873656.597705_373.1 2021-09-17T10:14:22.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:22.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:22.181 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:22.411 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:14:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:14:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:14:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:14:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:14:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:14:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:14:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:14:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:14:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873656.362432_367.1", 2021-09-17T10:14:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:14:16.362432Z", 2021-09-17T10:14:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:14:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:14:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:14:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:14:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:14:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:14:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:14:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:14:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:14:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:14:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:22.427 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:14:22.428 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873656.362432_367.1 2021-09-17T10:14:27.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:27.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:27.430 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:27.472 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:27.473 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:27.595 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:14:27.596 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:14:27.596 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:14:27.596 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:14:27.596 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:14:27.597 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:27.597 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:27.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:27.597 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:14:27.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:27.598 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:14:27.598 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:27.598 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:27.599 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:27.599 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:27.599 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:14:27.599 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:27.600 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:27.600 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:27.600 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:27.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:27.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:27.601 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:14:27.601 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:27.601 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:27.601 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873656.806856_380.1", 2021-09-17T10:14:27.602 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:27.602 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:27.602 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:27.603 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:14:16.806856Z", 2021-09-17T10:14:27.603 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:27.603 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:27.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:27.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:27.604 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:14:27.604 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:27.604 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:27.605 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:14:27.605 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:27.605 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:27.605 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:27.605 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:14:27.606 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:27.606 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:27.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:27.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:14:27.607 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:14:27.607 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:14:27.607 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:14:27.607 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:14:27.608 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:14:27.608 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:14:27.608 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:14:27.608 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:14:27.609 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:14:27.609 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:27.609 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:27.609 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:14:27.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:14:27.610 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:14:27.610 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:14:27.610 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:14:27.610 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:14:27.611 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:14:27.612 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:14:27.613 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:14:27.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:27.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:27.615 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:27.666 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:27.667 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:27.757 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:14:27.757 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:14:27.758 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:14:27.758 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:14:27.758 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:14:27.759 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:27.759 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:27.759 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:27.759 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:14:27.759 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:27.760 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:14:27.760 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:27.760 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:27.760 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:27.761 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:27.761 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:14:27.761 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:27.761 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:27.762 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:27.762 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:27.762 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:27.762 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:27.763 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:14:27.763 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:27.763 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:27.763 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:14:27.763 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:27.764 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:27.764 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:27.764 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:14:27.764 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:27.765 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:27.765 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:27.765 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:27.765 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:14:27.765 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:27.766 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:27.766 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:27.766 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:27.766 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:27.766 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:27.767 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:27.767 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:27.767 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:27.767 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:27.768 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:27.768 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:14:27.768 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:27.768 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:27.768 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:27.769 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:27.769 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:27.769 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:27.769 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:27.770 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:27.770 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:27.770 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:27.770 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:14:27.771 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:27.771 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:14:27.771 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:14:27.771 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:14:27.771 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:27.772 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:14:27.773 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:14:32.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:32.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:32.776 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:32.830 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:32.830 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:32.915 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:14:32.916 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:14:32.916 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:14:32.916 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:14:32.916 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:14:32.916 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:32.917 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:32.917 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:32.917 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:14:32.917 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:32.918 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:14:32.918 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:32.918 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:32.918 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:32.918 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:32.919 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:14:32.919 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:32.919 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:32.919 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:32.920 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:32.920 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:32.920 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:32.920 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:14:32.920 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:32.921 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:32.921 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:14:32.921 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:32.921 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:32.921 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:32.922 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:14:32.922 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:32.922 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:32.922 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:32.923 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:32.923 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:14:32.923 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:32.923 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:32.923 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:32.924 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:32.924 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:32.924 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:32.924 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:32.924 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:32.925 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:32.925 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:32.925 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:32.925 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:14:32.925 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:32.926 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:32.926 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:32.926 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:32.926 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:32.926 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:32.927 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:32.927 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:32.927 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:32.927 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:32.928 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:14:32.928 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:32.928 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:14:32.928 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:14:32.928 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:14:32.929 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:32.929 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:14:32.930 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.922243_318.1 2021-09-17T10:14:37.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:37.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:37.933 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:38.048 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:38.049 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:38.323 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:14:38.323 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:14:38.324 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:14:38.324 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:14:38.324 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:14:38.324 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:38.325 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:38.325 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:38.325 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:14:38.325 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:38.325 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:14:38.326 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:38.326 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:38.326 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:38.326 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:38.327 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:14:38.327 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:38.327 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:38.328 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:38.328 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:38.328 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:38.329 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:38.329 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:14:38.329 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:38.329 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:38.329 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:14:38.330 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:38.330 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:38.330 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:38.330 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:14:38.331 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:38.331 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:38.331 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:38.331 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:38.332 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:14:38.332 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:38.332 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:38.332 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:38.333 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:38.333 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:38.333 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:38.333 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:38.333 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:38.334 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:38.334 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:38.334 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:38.334 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:14:38.335 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:38.335 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:38.336 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:38.336 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:38.336 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:38.337 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:38.337 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:38.337 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:38.338 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:38.338 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:38.339 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:14:38.339 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:38.340 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:14:38.340 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:14:38.340 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:14:38.341 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:38.341 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:14:38.343 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.946914_319.1 2021-09-17T10:14:43.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:43.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:43.346 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:43.430 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:43.430 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:43.761 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:14:43.762 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:14:43.762 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:14:43.762 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:14:43.763 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:14:43.763 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:43.763 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:43.763 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:43.763 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:14:43.764 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:43.764 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:14:43.764 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:43.764 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:43.765 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:43.765 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:43.765 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:14:43.765 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:43.766 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:43.766 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:43.766 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:43.766 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:43.766 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:43.767 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:14:43.767 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:43.767 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:43.767 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:14:43.767 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:43.768 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:43.768 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:43.768 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:14:43.768 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:43.769 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:43.769 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:43.769 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:43.769 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:14:43.770 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:43.770 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:43.770 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:43.770 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:43.771 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:43.771 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:43.771 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:43.771 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:43.771 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:43.772 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:43.772 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:43.772 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:14:43.772 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:43.773 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:43.773 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:43.773 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:43.773 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:43.773 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:43.774 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:43.774 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:43.774 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:43.774 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:43.775 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:14:43.775 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:43.775 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:14:43.775 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:14:43.775 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:14:43.776 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:43.776 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:14:43.777 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.946914_319.1 2021-09-17T10:14:48.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:48.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:48.780 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:48.858 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:48.858 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:49.247 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:14:49.247 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:14:49.247 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:14:49.248 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:14:49.248 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:14:49.248 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:49.248 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:49.249 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:49.249 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:14:49.249 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:49.249 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:14:49.249 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:49.250 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:49.250 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:49.250 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:49.250 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:14:49.251 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:49.251 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:49.251 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:49.251 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:49.251 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:49.252 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:49.252 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:14:49.252 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:49.252 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:49.252 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:14:49.253 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:49.253 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:49.253 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:49.253 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:14:49.254 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:49.254 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:49.254 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:49.254 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:49.254 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:14:49.255 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:49.255 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:49.255 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:49.255 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:49.255 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:49.256 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:49.256 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:49.256 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:49.256 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:49.256 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:49.257 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:49.257 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:14:49.257 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:49.257 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:49.258 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:49.258 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:49.258 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:49.258 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:49.258 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:49.259 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:49.259 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:49.259 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:49.259 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:14:49.260 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:49.260 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:14:49.260 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:14:49.260 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:14:49.260 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:49.261 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:14:49.261 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.009924_325.1 2021-09-17T10:14:54.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:54.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:54.264 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:54.337 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:54.338 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:14:54.743 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:14:54.743 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:14:54.743 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:14:54.744 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:14:54.744 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:14:54.744 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:14:54.744 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:54.745 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:54.745 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:14:54.745 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:54.745 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:14:54.745 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:54.746 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:54.747 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:14:54.747 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:54.747 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:14:54.747 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:54.747 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:14:54.748 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:54.748 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:54.748 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:54.748 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:54.748 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:14:54.749 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:54.749 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:54.749 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:14:54.750 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:54.750 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:54.751 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:54.751 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:14:54.751 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:54.751 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:54.752 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:54.752 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:54.752 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:14:54.752 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:54.752 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:54.753 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:54.753 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:54.753 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:54.753 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:54.754 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:54.754 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:54.754 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:54.754 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:54.755 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:14:54.755 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:14:54.755 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:14:54.755 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:14:54.755 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:14:54.756 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:14:54.756 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:14:54.756 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:14:54.756 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:14:54.757 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:14:54.757 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:54.757 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:54.757 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:14:54.757 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:14:54.758 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:14:54.758 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:14:54.758 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:14:54.758 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:14:54.759 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:14:54.760 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.922243_318.1 2021-09-17T10:14:59.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:59.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:14:59.763 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:14:59.805 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:14:59.805 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:15:00.041 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:15:00.041 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:15:00.041 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:15:00.042 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:15:00.042 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:15:00.042 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:15:00.043 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:00.043 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:00.043 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:15:00.043 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:00.044 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:15:00.044 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:00.044 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:00.044 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:15:00.044 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:00.045 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:15:00.045 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:00.045 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:15:00.045 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:00.046 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:00.046 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:00.046 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:00.046 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:15:00.047 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:00.047 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:00.047 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:15:00.047 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:00.048 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:00.048 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:00.048 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:15:00.049 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:00.049 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:00.049 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:00.050 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:00.050 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:15:00.050 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:00.050 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:00.051 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:00.051 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:00.051 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:00.051 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:00.052 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:00.052 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:00.052 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:00.052 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:00.052 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:00.053 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:15:00.053 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:00.053 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:00.053 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:00.054 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:00.054 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:00.054 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:00.054 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:00.055 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:00.055 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:00.055 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:00.055 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:15:00.055 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:00.056 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:15:00.056 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:15:00.056 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:15:00.056 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:00.057 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:15:00.058 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.952206_320.1 2021-09-17T10:15:05.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:05.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:05.061 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:15:05.104 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:15:05.105 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:15:05.213 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:15:05.213 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:15:05.213 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:15:05.214 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:15:05.214 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:15:05.214 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:15:05.214 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:05.215 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:05.215 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:15:05.215 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:05.215 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:15:05.216 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:05.216 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:05.216 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:15:05.216 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:05.216 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:15:05.217 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:05.217 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:15:05.217 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:05.217 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:05.218 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:05.218 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:05.218 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:15:05.218 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:05.218 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:05.219 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.014731_326.1", 2021-09-17T10:15:05.219 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:05.219 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:05.219 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:05.220 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.014731Z", 2021-09-17T10:15:05.220 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:05.220 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:05.220 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:05.220 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:05.221 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:15:05.221 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:05.221 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:05.221 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:05.222 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:05.222 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:05.222 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:05.222 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:05.222 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:05.223 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:05.223 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:05.223 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:05.223 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:15:05.224 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:05.224 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:05.224 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:05.224 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:05.225 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:05.225 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:05.225 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:05.225 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:05.225 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:05.226 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:05.226 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:15:05.226 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:05.226 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:15:05.227 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:15:05.227 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:15:05.227 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:05.227 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:15:05.229 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.014731_326.1 2021-09-17T10:15:10.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:10.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:10.231 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:15:10.266 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:15:10.266 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:15:10.362 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:15:10.363 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:15:10.363 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:15:10.363 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:15:10.363 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:15:10.364 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:15:10.364 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:10.364 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:10.364 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:15:10.364 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:10.365 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:15:10.365 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:10.365 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:10.365 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:15:10.365 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:10.366 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:15:10.366 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:10.366 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:15:10.366 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:10.367 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:10.367 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:10.367 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:10.367 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:15:10.368 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:10.368 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:10.368 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.007158_324.1", 2021-09-17T10:15:10.368 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:10.368 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:10.369 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:10.369 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.007158Z", 2021-09-17T10:15:10.369 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:10.369 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:10.370 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:10.370 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:10.370 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:15:10.370 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:10.370 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:10.371 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:10.371 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:10.371 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:10.371 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:10.372 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:10.372 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:10.372 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:10.372 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:10.372 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:10.373 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:15:10.373 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:10.373 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:10.373 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:10.374 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:10.374 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:10.374 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:10.374 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:10.375 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:10.375 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:10.375 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:10.375 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:15:10.375 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:10.376 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:15:10.376 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:15:10.376 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:15:10.376 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:10.376 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:15:10.378 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.007158_324.1 2021-09-17T10:15:15.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:15.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:15.380 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:15:15.445 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:15:15.446 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:15:15.804 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:15:15.804 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:15:15.805 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:15:15.805 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:15:15.805 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:15:15.805 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:15:15.806 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:15.806 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:15.806 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:15:15.806 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:15.807 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:15:15.807 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:15.807 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:15.807 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:15:15.808 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:15.808 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:15:15.808 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:15.808 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:15:15.809 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:15.809 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:15.809 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:15.809 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:15.810 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:15:15.810 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:15.810 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:15.810 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:15:15.811 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:15.811 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:15.811 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:15.811 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:15:15.811 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:15.812 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:15.812 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:15.812 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:15.812 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:15:15.813 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:15.813 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:15.813 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:15.813 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:15.814 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:15.814 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:15.814 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:15.814 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:15.814 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:15.815 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:15.815 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:15.815 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:15:15.815 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:15.816 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:15.816 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:15.816 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:15.816 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:15.816 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:15.817 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:15.817 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:15.817 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:15.817 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:15.817 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:15:15.818 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:15.818 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:15:15.818 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:15:15.818 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:15:15.819 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:15.819 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:15:15.819 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.952206_320.1 2021-09-17T10:15:20.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:20.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:20.822 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:15:20.868 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:15:20.869 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:15:21.278 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:15:21.279 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:15:21.279 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:15:21.279 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:15:21.279 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:15:21.280 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:15:21.280 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:21.280 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:21.280 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:15:21.281 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:21.281 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:15:21.281 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:21.282 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:21.282 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:15:21.282 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:21.282 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:15:21.283 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:21.283 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:15:21.283 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:21.283 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:21.284 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:21.284 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:21.284 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:15:21.284 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:21.285 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:21.285 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:15:21.285 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:21.285 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:21.286 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:21.286 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:15:21.286 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:21.286 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:21.287 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:21.287 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:21.287 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:15:21.287 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:21.288 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:21.288 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:21.288 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:21.288 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:21.289 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:21.289 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:21.289 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:21.289 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:21.290 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:21.290 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:21.290 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:15:21.290 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:21.291 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:21.291 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:21.291 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:21.291 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:21.292 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:21.292 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:21.292 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:21.292 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:21.293 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:21.293 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:15:21.293 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:21.293 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:15:21.294 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:15:21.294 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:15:21.294 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:21.295 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:15:21.296 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:15:26.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:26.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:26.299 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:15:26.337 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:15:26.338 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:15:26.453 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:15:26.453 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:15:26.453 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:15:26.454 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:15:26.454 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:15:26.454 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:15:26.454 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:26.455 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:26.455 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:15:26.455 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:26.455 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:15:26.456 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:26.456 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:26.456 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:15:26.456 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:26.457 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:15:26.457 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:26.457 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:15:26.457 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:26.458 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:26.458 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:26.458 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:26.458 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:15:26.459 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:26.459 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:26.459 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.959198_322.1", 2021-09-17T10:15:26.459 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:26.459 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:26.460 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:26.460 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.959198Z", 2021-09-17T10:15:26.460 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:26.460 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:26.460 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:26.461 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:26.461 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:15:26.461 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:26.461 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:26.461 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:26.462 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:26.462 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:26.462 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:26.462 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:26.463 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:26.463 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:26.463 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:26.463 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:26.464 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:15:26.464 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:26.464 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:26.464 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:26.464 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:26.465 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:26.465 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:26.465 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:26.465 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:26.466 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:26.466 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:26.466 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:15:26.466 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:26.467 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:15:26.467 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:15:26.467 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:15:26.467 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:26.468 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:15:26.469 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.959198_322.1 2021-09-17T10:15:31.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:31.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:31.472 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:15:31.514 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:15:31.515 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:15:31.633 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:15:31.634 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:15:31.634 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:15:31.634 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:15:31.634 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:15:31.635 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:15:31.635 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:31.635 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:31.635 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:15:31.636 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:31.636 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:15:31.636 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:31.636 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:31.636 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:15:31.637 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:31.637 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:15:31.637 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:31.637 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:15:31.638 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:31.638 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:31.638 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:31.638 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:31.639 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:15:31.639 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:31.639 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:31.639 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:15:31.639 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:31.640 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:31.640 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:31.640 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:15:31.640 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:31.641 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:31.641 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:31.641 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:31.641 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:15:31.641 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:31.642 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:31.642 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:31.642 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:31.642 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:31.643 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:31.643 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:31.643 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:31.643 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:31.643 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:31.644 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:31.644 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:15:31.644 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:31.644 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:31.645 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:31.645 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:31.645 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:31.645 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:31.646 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:31.646 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:31.646 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:31.646 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:31.647 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:15:31.647 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:31.647 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:15:31.647 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:15:31.647 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:15:31.648 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:31.648 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:15:31.649 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.946914_319.1 2021-09-17T10:15:36.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:36.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:36.652 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:15:36.697 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:15:36.697 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:15:36.785 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:15:36.785 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:15:36.785 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:15:36.786 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:15:36.786 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:15:36.786 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:15:36.786 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:36.787 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:36.787 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:15:36.787 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:36.787 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:15:36.787 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:36.788 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:36.788 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:15:36.788 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:36.788 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:15:36.788 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:36.789 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:15:36.789 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:36.789 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:36.789 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:36.790 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:36.790 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:15:36.790 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:36.790 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:36.790 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:15:36.791 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:36.791 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:36.791 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:36.791 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:15:36.792 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:36.792 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:36.792 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:36.792 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:36.793 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:15:36.793 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:36.793 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:36.793 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:36.793 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:36.794 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:36.794 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:36.794 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:36.794 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:36.795 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:36.795 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:36.795 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:36.795 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:15:36.795 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:36.796 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:36.796 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:36.796 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:36.796 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:36.796 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:36.797 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:36.797 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:36.797 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:36.797 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:36.797 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:15:36.798 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:36.798 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:15:36.798 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:15:36.798 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:15:36.799 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:36.799 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:15:36.800 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.946914_319.1 2021-09-17T10:15:41.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:41.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:41.802 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:15:41.890 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:15:41.890 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:15:42.232 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:15:42.232 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:15:42.233 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:15:42.233 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:15:42.233 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:15:42.233 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:15:42.234 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:42.234 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:42.234 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:15:42.234 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:42.235 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:15:42.235 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:42.235 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:42.235 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:15:42.236 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:42.236 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:15:42.236 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:42.236 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:15:42.237 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:42.237 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:42.237 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:42.237 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:42.237 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:15:42.238 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:42.238 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:42.238 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:15:42.238 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:42.239 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:42.239 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:42.239 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:15:42.239 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:42.239 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:42.240 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:42.240 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:42.240 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:15:42.241 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:42.241 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:42.241 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:42.241 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:42.241 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:42.242 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:42.242 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:42.242 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:42.242 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:42.243 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:42.243 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:42.243 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:15:42.243 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:42.244 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:42.244 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:42.244 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:42.244 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:42.244 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:42.245 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:42.245 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:42.245 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:42.245 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:42.246 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:15:42.246 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:42.246 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:15:42.246 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:15:42.246 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:15:42.247 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:42.247 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:15:42.247 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.009924_325.1 2021-09-17T10:15:47.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:47.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:47.250 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:15:47.320 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:15:47.320 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:15:47.769 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:15:47.769 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:15:47.769 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:15:47.770 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:15:47.770 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:15:47.770 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:15:47.770 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:47.771 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:47.771 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:15:47.771 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:47.771 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:15:47.772 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:47.772 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:47.772 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:15:47.772 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:47.772 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:15:47.773 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:47.773 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:15:47.773 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:47.773 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:47.774 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:47.774 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:47.774 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:15:47.774 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:47.775 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:47.775 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:15:47.775 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:47.775 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:47.775 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:47.776 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:15:47.776 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:47.776 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:47.776 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:47.777 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:47.777 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:15:47.777 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:47.777 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:47.778 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:47.778 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:47.778 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:47.778 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:47.779 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:47.779 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:47.779 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:47.779 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:47.779 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:47.780 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:15:47.780 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:47.780 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:47.780 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:47.781 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:47.781 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:47.781 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:47.781 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:47.782 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:47.782 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:47.782 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:47.782 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:15:47.782 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:47.783 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:15:47.783 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:15:47.783 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:15:47.783 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:47.784 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:15:47.785 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.922243_318.1 2021-09-17T10:15:52.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:52.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:52.788 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:15:52.871 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:15:52.871 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:15:53.272 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:15:53.273 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:15:53.273 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:15:53.273 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:15:53.273 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:15:53.274 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:15:53.274 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:53.274 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:53.274 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:15:53.275 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:53.275 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:15:53.275 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:53.275 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:53.276 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:15:53.276 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:53.276 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:15:53.276 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:53.276 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:15:53.277 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:53.277 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:53.277 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:53.277 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:53.278 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:15:53.278 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:53.278 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:53.278 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.038405_327.1", 2021-09-17T10:15:53.278 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:53.279 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:53.279 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:53.279 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.038405Z", 2021-09-17T10:15:53.279 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:53.279 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:53.280 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:53.280 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:53.280 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:15:53.280 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:53.281 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:53.281 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:53.281 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:53.281 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:53.281 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:53.281 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:53.282 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:53.282 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:53.282 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:53.282 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:53.283 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:15:53.283 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:53.283 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:53.283 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:53.284 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:53.284 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:53.284 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:53.284 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:53.284 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:53.285 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:53.285 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:53.285 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:15:53.285 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:53.286 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:15:53.286 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:15:53.286 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:15:53.286 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:53.287 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:15:53.288 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.038405_327.1 2021-09-17T10:15:58.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:58.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:15:58.290 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:15:58.334 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:15:58.335 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:15:58.446 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:15:58.447 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:15:58.447 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:15:58.447 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:15:58.448 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:15:58.448 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:15:58.448 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:58.449 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:58.449 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:15:58.449 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:58.449 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:15:58.450 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:58.450 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:58.450 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:15:58.450 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:58.451 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:15:58.451 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:58.451 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:15:58.451 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:58.451 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:58.452 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:58.452 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:58.452 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:15:58.453 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:58.453 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:58.453 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.256150_335.1", 2021-09-17T10:15:58.453 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:58.454 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:58.454 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:58.454 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.256150Z", 2021-09-17T10:15:58.454 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:58.454 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:58.455 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:58.455 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:58.455 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:15:58.455 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:58.456 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:58.456 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:58.456 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:58.456 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:58.457 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:58.457 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:58.457 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:58.457 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:58.458 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:58.458 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:15:58.458 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:15:58.458 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:15:58.459 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:15:58.459 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:15:58.459 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:15:58.459 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:15:58.460 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:15:58.460 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:15:58.460 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:15:58.460 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:58.461 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:58.461 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:15:58.461 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:15:58.461 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:15:58.461 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:15:58.462 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:15:58.462 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:15:58.462 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:15:58.463 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.256150_335.1 2021-09-17T10:16:03.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:03.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:03.466 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:16:03.507 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:16:03.507 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:16:03.624 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:16:03.624 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:16:03.624 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:16:03.624 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:16:03.625 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:16:03.625 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:16:03.625 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:03.625 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:03.626 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:16:03.626 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:03.626 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:16:03.626 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:03.626 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:03.627 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:16:03.627 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:03.627 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:16:03.627 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:03.627 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:16:03.628 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:03.628 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:03.628 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:03.628 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:03.628 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:16:03.629 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:03.629 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:03.629 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.263983_336.1", 2021-09-17T10:16:03.629 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:03.630 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:03.630 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:03.630 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.263983Z", 2021-09-17T10:16:03.630 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:03.630 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:03.631 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:03.631 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:03.631 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:16:03.631 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:03.631 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:03.632 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:03.632 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:03.632 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:03.632 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:03.632 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:03.633 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:03.633 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:03.633 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:03.633 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:03.633 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:16:03.634 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:03.634 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:03.634 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:03.634 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:03.634 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:03.635 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:03.635 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:03.635 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:03.635 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:03.635 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:03.636 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:16:03.636 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:03.636 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:16:03.636 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:16:03.637 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:16:03.637 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:03.637 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:16:03.638 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.263983_336.1 2021-09-17T10:16:08.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:08.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:08.640 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:16:08.706 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:16:08.707 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:16:08.850 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:16:08.851 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:16:08.851 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:16:08.851 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:16:08.851 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:16:08.852 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:16:08.852 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:08.852 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:08.852 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:16:08.853 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:08.853 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:16:08.853 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:08.853 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:08.853 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:16:08.854 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:08.854 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:16:08.854 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:08.854 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:16:08.855 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:08.855 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:08.855 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:08.855 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:08.856 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:16:08.856 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:08.856 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:08.856 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.105095_330.1", 2021-09-17T10:16:08.856 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:08.857 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:08.857 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:08.857 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.105095Z", 2021-09-17T10:16:08.857 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:08.858 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:08.858 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:08.858 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:08.858 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:16:08.859 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:08.859 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:08.859 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:08.859 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:08.860 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:08.860 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:08.860 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:08.860 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:08.860 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:08.861 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:08.861 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:08.861 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:16:08.861 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:08.862 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:08.862 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:08.862 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:08.862 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:08.862 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:08.863 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:08.863 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:08.863 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:08.863 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:08.864 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:16:08.864 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:08.864 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:16:08.864 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:16:08.865 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:16:08.865 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:08.865 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:16:08.866 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.105095_330.1 2021-09-17T10:16:13.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:13.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:13.869 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:16:13.938 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:16:13.938 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:16:14.360 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:16:14.360 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:16:14.360 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:16:14.361 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:16:14.361 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:16:14.361 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:16:14.361 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:14.361 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:14.362 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:16:14.362 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:14.362 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:16:14.362 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:14.363 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:14.363 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:16:14.363 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:14.363 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:16:14.364 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:14.364 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:16:14.364 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:14.364 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:14.364 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:14.365 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:14.365 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:16:14.365 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:14.365 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:14.365 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:16:14.366 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:14.366 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:14.366 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:14.366 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:16:14.367 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:14.367 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:14.367 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:14.367 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:14.368 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:16:14.368 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:14.368 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:14.369 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:14.369 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:14.369 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:14.369 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:14.369 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:14.370 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:14.370 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:14.370 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:14.370 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:14.371 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:16:14.371 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:14.371 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:14.371 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:14.371 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:14.372 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:14.372 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:14.372 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:14.372 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:14.373 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:14.373 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:14.373 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:16:14.373 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:14.373 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:16:14.374 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:16:14.374 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:16:14.374 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:14.374 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:16:14.375 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.922243_318.1 2021-09-17T10:16:19.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:19.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:19.378 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:16:19.420 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:16:19.420 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:16:19.611 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:16:19.611 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:16:19.611 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:16:19.612 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:16:19.612 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:16:19.612 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:16:19.612 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:19.612 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:19.613 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:16:19.613 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:19.613 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:16:19.613 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:19.613 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:19.614 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:16:19.614 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:19.614 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:16:19.614 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:19.615 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:16:19.615 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:19.615 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:19.615 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:19.615 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:19.616 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:16:19.616 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:19.616 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:19.616 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:16:19.616 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:19.617 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:19.617 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:19.617 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:16:19.617 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:19.618 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:19.618 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:19.618 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:19.618 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:16:19.619 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:19.619 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:19.619 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:19.619 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:19.619 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:19.620 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:19.620 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:19.620 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:19.620 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:19.620 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:19.621 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:19.621 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:16:19.621 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:19.621 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:19.622 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:19.622 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:19.622 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:19.622 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:19.622 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:19.623 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:19.623 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:19.623 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:19.623 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:16:19.624 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:19.624 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:16:19.624 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:16:19.624 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:16:19.624 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:19.625 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:16:19.626 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:16:24.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:24.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:24.628 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:16:24.671 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:16:24.671 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:16:24.786 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:16:24.786 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:16:24.787 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:16:24.787 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:16:24.787 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:16:24.787 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:16:24.788 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:24.788 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:24.788 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:16:24.789 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:24.789 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:16:24.789 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:24.789 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:24.790 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:16:24.790 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:24.790 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:16:24.790 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:24.791 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:16:24.791 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:24.791 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:24.791 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:24.792 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:24.792 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:16:24.792 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:24.792 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:24.793 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:16:24.793 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:24.793 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:24.793 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:24.794 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:16:24.794 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:24.794 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:24.794 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:24.795 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:24.795 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:16:24.795 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:24.795 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:24.796 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:24.796 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:24.796 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:24.796 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:24.797 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:24.797 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:24.797 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:24.797 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:24.798 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:24.798 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:16:24.798 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:24.798 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:24.798 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:24.799 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:24.799 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:24.799 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:24.799 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:24.800 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:24.800 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:24.800 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:24.800 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:16:24.800 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:24.801 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:16:24.801 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:16:24.801 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:16:24.801 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:24.801 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:16:24.803 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.922243_318.1 2021-09-17T10:16:29.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:29.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:29.806 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:16:29.848 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:16:29.849 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:16:29.959 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:16:29.960 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:16:29.960 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:16:29.960 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:16:29.961 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:16:29.961 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:16:29.961 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:29.961 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:29.962 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:16:29.962 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:29.962 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:16:29.962 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:29.963 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:29.963 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:16:29.963 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:29.963 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:16:29.964 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:29.964 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:16:29.964 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:29.965 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:29.965 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:29.965 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:29.965 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:16:29.966 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:29.966 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:29.966 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.256150_335.1", 2021-09-17T10:16:29.966 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:29.967 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:29.967 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:29.967 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.256150Z", 2021-09-17T10:16:29.968 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:29.968 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:29.968 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:29.968 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:29.968 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:16:29.969 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:29.969 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:29.969 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:29.970 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:29.970 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:29.970 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:29.970 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:29.971 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:29.971 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:29.971 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:29.971 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:29.972 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:16:29.972 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:29.972 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:29.972 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:29.973 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:29.973 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:29.973 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:29.973 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:29.974 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:29.974 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:29.974 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:29.975 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:16:29.975 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:29.975 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:16:29.975 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:16:29.976 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:16:29.976 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:29.976 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:16:29.977 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.256150_335.1 2021-09-17T10:16:34.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:34.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:34.980 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:16:35.058 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:16:35.058 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:16:35.302 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:16:35.303 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:16:35.303 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:16:35.303 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:16:35.304 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:16:35.304 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:16:35.304 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:35.304 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:35.305 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:16:35.305 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:35.305 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:16:35.305 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:35.306 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:35.306 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:16:35.306 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:35.306 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:16:35.306 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:35.307 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:16:35.307 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:35.307 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:35.307 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:35.308 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:35.308 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:16:35.308 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:35.308 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:35.309 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:16:35.309 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:35.309 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:35.309 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:35.309 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:16:35.310 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:35.310 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:35.310 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:35.310 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:35.311 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:16:35.311 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:35.311 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:35.311 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:35.311 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:35.312 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:35.312 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:35.312 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:35.312 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:35.312 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:35.313 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:35.313 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:35.313 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:16:35.313 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:35.314 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:35.314 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:35.314 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:35.314 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:35.315 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:35.315 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:35.315 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:35.315 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:35.315 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:35.316 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:16:35.316 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:35.316 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:16:35.316 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:16:35.317 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:16:35.317 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:35.317 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:16:35.318 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:16:40.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:40.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:40.321 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:16:40.415 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:16:40.416 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:16:40.793 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:16:40.793 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:16:40.794 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:16:40.794 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:16:40.794 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:16:40.794 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:16:40.795 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:40.795 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:40.795 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:16:40.795 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:40.795 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:16:40.796 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:40.796 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:40.796 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:16:40.796 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:40.797 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:16:40.797 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:40.797 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:16:40.797 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:40.797 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:40.798 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:40.798 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:40.798 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:16:40.798 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:40.799 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:40.799 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.014731_326.1", 2021-09-17T10:16:40.799 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:40.799 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:40.800 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:40.800 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.014731Z", 2021-09-17T10:16:40.800 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:40.800 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:40.800 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:40.801 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:40.801 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:16:40.801 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:40.801 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:40.801 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:40.802 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:40.802 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:40.802 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:40.802 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:40.803 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:40.803 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:40.803 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:40.803 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:40.803 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:16:40.804 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:40.804 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:40.804 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:40.804 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:40.804 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:40.805 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:40.805 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:40.805 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:40.805 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:40.806 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:40.806 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:16:40.806 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:40.806 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:16:40.807 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:16:40.807 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:16:40.807 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:40.807 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:16:40.808 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.014731_326.1 2021-09-17T10:16:45.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:45.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:45.811 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:16:45.851 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:16:45.852 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:16:45.976 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:16:45.977 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:16:45.977 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:16:45.977 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:16:45.977 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:16:45.978 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:16:45.978 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:45.978 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:45.978 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:16:45.979 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:45.979 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:16:45.979 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:45.979 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:45.980 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:16:45.980 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:45.980 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:16:45.980 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:45.980 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:16:45.981 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:45.981 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:45.981 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:45.981 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:45.982 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:16:45.982 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:45.982 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:45.982 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.014731_326.1", 2021-09-17T10:16:45.983 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:45.983 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:45.983 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:45.983 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.014731Z", 2021-09-17T10:16:45.983 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:45.984 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:45.984 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:45.984 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:45.984 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:16:45.985 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:45.985 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:45.985 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:45.985 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:45.986 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:45.986 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:45.986 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:45.986 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:45.986 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:45.987 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:45.987 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:45.987 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:16:45.987 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:45.988 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:45.988 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:45.988 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:45.988 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:45.989 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:45.989 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:45.989 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:45.989 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:45.989 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:45.990 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:16:45.990 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:45.990 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:16:45.990 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:16:45.991 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:16:45.991 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:45.991 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:16:45.992 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.014731_326.1 2021-09-17T10:16:50.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:50.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:50.995 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:16:51.037 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:16:51.037 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:16:51.149 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:16:51.150 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:16:51.150 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:16:51.150 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:16:51.150 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:16:51.151 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:16:51.151 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:51.151 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:51.151 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:16:51.151 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:51.152 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:16:51.152 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:51.152 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:51.152 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:16:51.152 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:51.153 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:16:51.153 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:51.153 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:16:51.153 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:51.154 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:51.154 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:51.154 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:51.154 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:16:51.155 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:51.155 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:51.155 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.038405_327.1", 2021-09-17T10:16:51.155 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:51.155 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:51.156 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:51.156 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.038405Z", 2021-09-17T10:16:51.156 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:51.156 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:51.156 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:51.157 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:51.157 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:16:51.157 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:51.157 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:51.157 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:51.158 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:51.158 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:51.158 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:51.158 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:51.158 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:51.159 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:51.159 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:51.159 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:51.159 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:16:51.160 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:51.160 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:51.160 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:51.160 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:51.160 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:51.161 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:51.161 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:51.161 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:51.161 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:51.161 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:51.162 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:16:51.162 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:51.162 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:16:51.162 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:16:51.162 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:16:51.163 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:51.163 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:16:51.164 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.038405_327.1 2021-09-17T10:16:56.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:56.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:16:56.167 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:16:56.229 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:16:56.230 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:16:56.341 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:16:56.341 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:16:56.342 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:16:56.342 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:16:56.342 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:16:56.343 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:16:56.343 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:56.343 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:56.343 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:16:56.344 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:56.344 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:16:56.344 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:56.344 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:56.344 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:16:56.345 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:56.345 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:16:56.345 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:56.345 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:16:56.346 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:56.346 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:56.346 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:56.346 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:56.347 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:16:56.347 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:56.347 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:56.347 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.419688_344.1", 2021-09-17T10:16:56.347 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:56.348 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:56.348 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:56.348 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.419688Z", 2021-09-17T10:16:56.348 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:56.349 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:56.349 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:56.349 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:56.349 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:16:56.349 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:56.350 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:56.350 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:56.350 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:56.350 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:56.351 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:56.351 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:56.351 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:56.351 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:56.351 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:56.352 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:16:56.352 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:16:56.352 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:16:56.352 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:16:56.353 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:16:56.353 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:16:56.353 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:16:56.353 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:16:56.353 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:16:56.354 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:16:56.354 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:56.354 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:56.354 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:16:56.355 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:16:56.355 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:16:56.355 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:16:56.355 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:16:56.355 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:16:56.356 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:16:56.356 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.419688_344.1 2021-09-17T10:17:01.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:01.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:01.359 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:17:01.422 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:17:01.423 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:17:01.888 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:17:01.888 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:17:01.888 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:17:01.888 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:17:01.889 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:17:01.889 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:17:01.889 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:01.889 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:01.890 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:17:01.890 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:01.890 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:17:01.890 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:01.891 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:01.891 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:17:01.891 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:01.891 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:17:01.891 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:01.892 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:17:01.892 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:01.892 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:01.892 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:01.893 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:01.893 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:17:01.893 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:01.893 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:01.894 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:17:01.894 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:01.894 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:01.894 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:01.894 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:17:01.895 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:01.895 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:01.895 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:01.895 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:01.896 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:17:01.896 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:01.896 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:01.896 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:01.897 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:01.897 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:01.897 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:01.897 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:01.897 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:01.898 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:01.898 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:01.898 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:01.898 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:17:01.899 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:01.899 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:01.899 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:01.899 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:01.900 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:01.900 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:01.900 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:01.900 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:01.901 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:01.901 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:01.901 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:17:01.901 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:01.901 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:17:01.902 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:17:01.902 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:17:01.902 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:01.902 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:17:01.904 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:17:06.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:06.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:06.906 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:17:06.948 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:17:06.949 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:17:07.173 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:17:07.174 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:17:07.174 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:17:07.174 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:17:07.174 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:17:07.175 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:17:07.175 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:07.175 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:07.175 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:17:07.175 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:07.176 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:17:07.176 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:07.176 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:07.176 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:17:07.177 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:07.177 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:17:07.177 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:07.177 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:17:07.177 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:07.178 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:07.178 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:07.178 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:07.178 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:17:07.179 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:07.179 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:07.179 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.344112_340.1", 2021-09-17T10:17:07.179 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:07.180 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:07.180 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:07.180 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.344112Z", 2021-09-17T10:17:07.180 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:07.180 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:07.181 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:07.181 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:07.181 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:17:07.181 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:07.182 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:07.182 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:07.182 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:07.182 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:07.183 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:07.183 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:07.183 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:07.183 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:07.184 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:07.184 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:07.184 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:17:07.184 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:07.184 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:07.185 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:07.185 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:07.185 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:07.185 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:07.186 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:07.186 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:07.186 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:07.186 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:07.187 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:17:07.187 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:07.187 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:17:07.187 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:17:07.188 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:17:07.188 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:07.188 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:17:07.189 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.344112_340.1 2021-09-17T10:17:12.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:12.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:12.193 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:17:12.235 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:17:12.235 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:17:12.341 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:17:12.341 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:17:12.342 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:17:12.342 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:17:12.342 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:17:12.342 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:17:12.343 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:12.343 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:12.343 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:17:12.343 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:12.344 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:17:12.344 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:12.344 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:12.344 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:17:12.344 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:12.345 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:17:12.345 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:12.345 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:17:12.345 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:12.346 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:12.346 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:12.346 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:12.346 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:17:12.347 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:12.347 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:12.347 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.448543_346.1", 2021-09-17T10:17:12.347 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:12.348 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:12.348 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:12.348 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.448543Z", 2021-09-17T10:17:12.348 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:12.348 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:12.349 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:12.349 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:12.349 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:17:12.349 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:12.350 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:12.350 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:12.350 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:12.350 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:12.350 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:12.351 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:12.351 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:12.351 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:12.351 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:12.352 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:12.352 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:17:12.352 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:12.352 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:12.352 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:12.353 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:12.353 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:12.353 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:12.353 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:12.354 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:12.354 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:12.354 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:12.354 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:17:12.354 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:12.355 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:17:12.355 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:17:12.355 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:17:12.355 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:12.356 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:17:12.357 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.448543_346.1 2021-09-17T10:17:17.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:17.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:17.359 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:17:17.397 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:17:17.397 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:17:17.477 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:17:17.478 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:17:17.478 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:17:17.478 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:17:17.478 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:17:17.479 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:17:17.479 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:17.479 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:17.479 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:17:17.480 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:17.480 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:17:17.480 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:17.480 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:17.481 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:17:17.481 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:17.481 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:17:17.481 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:17.481 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:17:17.482 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:17.482 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:17.482 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:17.482 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:17.483 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:17:17.483 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:17.483 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:17.483 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:17:17.483 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:17.484 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:17.484 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:17.484 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:17:17.484 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:17.485 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:17.485 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:17.485 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:17.485 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:17:17.486 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:17.486 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:17.486 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:17.486 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:17.486 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:17.487 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:17.487 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:17.487 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:17.487 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:17.488 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:17.488 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:17.488 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:17:17.488 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:17.489 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:17.489 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:17.489 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:17.489 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:17.489 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:17.490 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:17.490 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:17.490 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:17.490 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:17.491 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:17:17.491 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:17.491 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:17:17.491 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:17:17.492 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:17:17.492 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:17.492 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:17:17.493 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.922243_318.1 2021-09-17T10:17:22.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:22.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:22.496 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:17:22.579 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:17:22.580 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:17:22.943 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:17:22.944 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:17:22.944 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:17:22.944 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:17:22.944 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:17:22.945 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:17:22.945 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:22.945 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:22.945 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:17:22.945 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:22.946 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:17:22.946 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:22.946 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:22.946 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:17:22.947 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:22.947 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:17:22.947 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:22.947 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:17:22.948 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:22.948 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:22.948 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:22.948 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:22.949 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:17:22.949 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:22.949 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:22.949 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.014731_326.1", 2021-09-17T10:17:22.949 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:22.950 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:22.950 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:22.950 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.014731Z", 2021-09-17T10:17:22.950 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:22.951 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:22.951 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:22.951 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:22.951 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:17:22.951 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:22.952 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:22.952 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:22.952 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:22.952 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:22.953 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:22.953 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:22.953 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:22.953 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:22.953 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:22.954 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:22.954 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:17:22.954 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:22.954 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:22.955 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:22.955 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:22.955 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:22.955 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:22.955 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:22.956 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:22.956 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:22.956 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:22.956 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:17:22.956 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:22.957 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:17:22.957 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:17:22.957 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:17:22.957 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:22.958 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:17:22.958 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.014731_326.1 2021-09-17T10:17:27.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:27.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:27.961 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:17:28.042 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:17:28.043 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:17:28.457 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:17:28.457 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:17:28.458 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:17:28.458 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:17:28.458 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:17:28.458 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:17:28.459 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:28.459 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:28.459 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:17:28.459 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:28.460 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:17:28.460 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:28.460 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:28.460 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:17:28.460 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:28.461 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:17:28.461 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:28.461 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:17:28.461 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:28.461 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:28.462 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:28.462 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:28.462 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:17:28.462 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:28.462 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:28.463 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.001301_323.1", 2021-09-17T10:17:28.463 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:28.463 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:28.463 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:28.464 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.001301Z", 2021-09-17T10:17:28.464 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:28.464 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:28.464 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:28.465 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:28.465 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:17:28.465 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:28.465 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:28.465 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:28.466 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:28.466 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:28.466 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:28.466 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:28.467 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:28.467 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:28.467 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:28.467 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:28.468 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:17:28.468 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:28.468 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:28.468 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:28.468 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:28.469 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:28.469 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:28.469 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:28.469 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:28.470 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:28.470 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:28.470 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:17:28.470 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:28.471 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:17:28.471 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:17:28.471 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:17:28.471 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:28.472 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:17:28.472 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.001301_323.1 2021-09-17T10:17:33.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:33.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:33.475 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:17:33.516 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:17:33.516 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:17:33.634 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:17:33.634 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:17:33.635 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:17:33.635 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:17:33.635 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:17:33.635 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:17:33.636 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:33.636 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:33.636 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:17:33.636 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:33.637 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:17:33.637 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:33.637 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:33.637 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:17:33.638 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:33.638 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:17:33.638 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:33.638 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:17:33.639 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:33.639 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:33.639 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:33.639 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:33.639 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:17:33.640 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:33.640 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:33.640 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.249312_334.1", 2021-09-17T10:17:33.640 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:33.641 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:33.641 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:33.641 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.249312Z", 2021-09-17T10:17:33.641 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:33.642 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:33.642 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:33.642 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:33.642 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:17:33.643 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:33.643 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:33.643 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:33.643 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:33.643 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:33.644 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:33.644 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:33.644 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:33.644 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:33.645 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:33.645 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:33.645 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:17:33.645 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:33.646 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:33.646 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:33.646 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:33.646 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:33.647 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:33.647 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:33.647 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:33.647 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:33.648 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:33.648 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:17:33.648 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:33.648 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:17:33.648 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:17:33.649 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:17:33.649 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:33.649 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:17:33.650 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.249312_334.1 2021-09-17T10:17:38.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:38.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:38.654 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:17:38.693 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:17:38.693 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:17:38.807 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:17:38.808 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:17:38.808 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:17:38.808 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:17:38.809 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:17:38.809 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:17:38.809 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:38.809 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:38.810 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:17:38.810 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:38.810 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:17:38.810 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:38.811 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:38.811 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:17:38.811 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:38.811 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:17:38.811 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:38.812 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:17:38.812 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:38.812 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:38.812 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:38.813 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:38.813 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:17:38.813 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:38.813 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:38.813 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:17:38.814 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:38.814 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:38.814 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:38.814 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:17:38.815 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:38.815 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:38.815 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:38.816 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:38.816 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:17:38.816 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:38.816 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:38.816 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:38.817 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:38.817 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:38.817 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:38.817 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:38.817 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:38.818 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:38.818 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:38.818 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:38.818 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:17:38.819 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:38.819 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:38.820 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:38.820 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:38.820 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:38.820 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:38.820 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:38.821 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:38.821 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:38.822 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:38.822 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:17:38.822 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:38.822 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:17:38.822 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:17:38.823 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:17:38.823 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:38.823 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:17:38.824 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.922243_318.1 2021-09-17T10:17:43.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:43.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:43.827 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:17:43.895 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:17:43.895 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:17:44.035 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:17:44.036 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:17:44.036 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:17:44.036 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:17:44.036 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:17:44.037 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:17:44.037 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:44.037 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:44.037 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:17:44.038 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:44.038 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:17:44.038 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:44.038 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:44.038 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:17:44.039 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:44.039 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:17:44.039 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:44.039 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:17:44.040 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:44.040 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:44.040 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:44.040 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:44.041 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:17:44.041 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:44.041 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:44.041 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:17:44.041 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:44.042 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:44.042 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:44.042 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:17:44.042 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:44.043 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:44.043 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:44.043 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:44.043 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:17:44.044 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:44.044 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:44.044 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:44.044 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:44.044 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:44.045 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:44.045 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:44.045 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:44.045 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:44.046 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:44.046 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:44.046 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:17:44.046 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:44.046 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:44.047 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:44.047 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:44.047 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:44.047 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:44.048 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:44.048 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:44.048 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:44.048 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:44.048 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:17:44.049 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:44.049 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:17:44.049 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:17:44.049 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:17:44.050 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:44.050 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:17:44.051 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.952206_320.1 2021-09-17T10:17:49.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:49.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:49.054 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:17:49.120 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:17:49.120 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:17:49.505 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:17:49.505 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:17:49.506 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:17:49.506 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:17:49.506 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:17:49.506 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:17:49.507 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:49.507 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:49.507 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:17:49.507 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:49.507 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:17:49.508 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:49.508 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:49.508 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:17:49.508 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:49.509 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:17:49.509 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:49.509 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:17:49.509 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:49.509 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:49.510 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:49.510 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:49.510 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:17:49.510 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:49.511 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:49.511 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.959198_322.1", 2021-09-17T10:17:49.511 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:49.511 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:49.511 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:49.512 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.959198Z", 2021-09-17T10:17:49.512 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:49.512 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:49.512 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:49.513 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:49.513 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:17:49.513 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:49.513 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:49.513 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:49.514 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:49.514 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:49.514 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:49.514 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:49.515 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:49.515 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:49.515 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:49.515 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:49.515 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:17:49.516 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:49.516 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:49.516 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:49.516 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:49.517 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:49.517 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:49.517 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:49.517 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:49.517 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:49.518 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:49.518 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:17:49.518 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:49.518 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:17:49.519 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:17:49.519 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:17:49.519 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:49.519 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:17:49.520 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.959198_322.1 2021-09-17T10:17:54.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:54.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:54.523 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:17:54.566 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:17:54.566 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:17:54.734 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:17:54.734 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:17:54.734 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:17:54.735 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:17:54.735 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:17:54.735 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:17:54.735 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:54.735 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:54.736 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:17:54.736 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:54.736 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:17:54.736 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:54.737 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:54.737 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:17:54.737 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:54.737 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:17:54.737 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:54.738 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:17:54.738 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:54.738 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:54.738 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:54.738 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:54.739 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:17:54.739 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:54.739 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:54.739 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:17:54.740 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:54.740 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:54.740 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:54.740 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:17:54.740 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:54.741 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:54.741 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:54.741 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:54.741 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:17:54.742 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:54.742 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:54.742 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:54.742 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:54.742 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:54.743 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:54.743 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:54.743 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:54.743 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:54.744 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:54.744 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:54.744 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:17:54.744 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:54.744 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:54.745 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:54.745 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:54.745 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:54.745 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:54.746 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:54.746 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:54.746 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:54.746 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:54.746 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:17:54.747 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:54.747 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:17:54.747 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:17:54.747 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:17:54.748 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:54.748 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:17:54.749 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:17:59.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:59.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:17:59.752 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:17:59.794 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:17:59.795 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:17:59.914 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:17:59.914 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:17:59.915 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:17:59.915 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:17:59.915 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:17:59.915 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:17:59.916 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:59.916 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:59.916 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:17:59.916 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:59.917 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:17:59.917 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:59.917 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:59.917 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:17:59.918 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:59.918 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:17:59.918 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:59.918 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:17:59.919 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:59.919 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:59.919 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:59.919 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:59.920 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:17:59.920 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:59.920 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:59.920 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:17:59.921 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:59.921 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:59.921 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:59.921 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:17:59.921 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:59.922 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:59.922 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:59.922 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:59.922 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:17:59.923 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:59.923 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:59.923 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:59.923 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:59.923 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:59.924 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:59.924 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:59.924 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:59.924 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:59.925 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:59.925 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:17:59.925 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:17:59.925 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:17:59.926 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:17:59.926 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:17:59.926 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:17:59.926 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:17:59.926 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:17:59.927 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:17:59.927 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:17:59.927 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:59.927 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:59.928 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:17:59.928 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:17:59.928 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:17:59.928 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:17:59.928 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:17:59.929 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:17:59.929 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:17:59.930 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:18:04.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:04.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:04.933 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:18:04.987 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:18:04.988 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:18:05.106 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:18:05.106 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:18:05.107 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:18:05.107 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:18:05.107 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:18:05.107 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:18:05.108 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:05.108 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:05.108 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:18:05.108 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:05.109 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:18:05.109 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:05.109 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:05.109 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:18:05.110 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:05.110 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:18:05.110 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:05.110 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:18:05.110 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:05.111 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:05.111 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:05.111 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:05.111 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:18:05.112 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:05.112 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:05.112 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:18:05.112 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:05.112 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:05.113 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:05.113 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:18:05.113 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:05.113 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:05.114 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:05.114 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:05.114 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:18:05.114 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:05.115 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:05.115 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:05.115 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:05.115 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:05.115 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:05.116 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:05.116 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:05.116 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:05.116 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:05.117 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:05.117 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:18:05.117 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:05.117 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:05.117 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:05.118 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:05.118 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:05.118 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:05.118 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:05.119 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:05.119 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:05.119 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:05.119 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:18:05.120 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:05.120 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:18:05.120 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:18:05.120 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:18:05.120 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:05.121 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:18:05.122 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.922243_318.1 2021-09-17T10:18:10.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:10.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:10.124 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:18:10.184 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:18:10.185 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:18:10.593 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:18:10.594 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:18:10.594 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:18:10.594 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:18:10.594 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:18:10.595 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:18:10.595 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:10.595 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:10.595 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:18:10.596 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:10.596 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:18:10.596 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:10.596 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:10.597 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:18:10.597 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:10.597 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:18:10.597 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:10.598 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:18:10.598 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:10.598 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:10.598 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:10.598 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:10.599 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:18:10.599 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:10.599 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:10.599 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:18:10.600 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:10.600 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:10.600 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:10.601 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:18:10.601 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:10.601 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:10.601 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:10.602 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:10.602 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:18:10.602 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:10.602 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:10.603 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:10.603 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:10.603 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:10.603 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:10.603 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:10.604 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:10.604 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:10.604 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:10.604 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:10.604 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:18:10.605 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:10.605 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:10.605 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:10.605 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:10.606 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:10.606 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:10.606 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:10.606 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:10.606 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:10.607 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:10.607 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:18:10.607 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:10.607 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:18:10.608 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:18:10.608 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:18:10.608 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:10.608 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:18:10.609 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:18:15.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:15.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:15.612 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:18:15.664 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:18:15.761 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:18:16.138 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:18:16.138 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:18:16.139 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:18:16.143 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:18:16.143 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:18:16.143 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:18:16.144 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:16.144 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:16.144 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:18:16.144 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:16.145 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:18:16.145 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:16.145 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:16.146 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:18:16.146 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:16.146 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:18:16.146 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:16.147 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:18:16.147 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:16.147 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:16.147 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:16.148 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:16.148 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:18:16.148 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:16.148 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:16.149 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:18:16.149 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:16.149 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:16.149 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:16.150 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:18:16.150 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:16.150 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:16.150 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:16.151 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:16.151 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:18:16.151 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:16.151 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:16.152 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:16.152 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:16.152 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:16.152 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:16.153 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:16.153 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:16.153 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:16.153 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:16.154 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:16.154 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:18:16.154 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:16.154 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:16.155 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:16.155 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:16.155 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:16.155 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:16.156 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:16.156 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:16.156 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:16.156 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:16.156 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:18:16.157 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:16.157 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:18:16.157 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:18:16.157 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:18:16.158 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:16.158 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:18:16.159 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.946914_319.1 2021-09-17T10:18:21.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:21.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:21.163 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:18:21.205 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:18:21.205 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:18:21.327 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:18:21.328 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:18:21.328 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:18:21.328 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:18:21.328 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:18:21.329 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:18:21.329 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:21.329 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:21.329 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:18:21.330 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:21.330 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:18:21.330 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:21.330 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:21.331 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:18:21.331 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:21.331 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:18:21.331 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:21.331 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:18:21.332 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:21.332 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:21.332 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:21.332 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:21.333 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:18:21.333 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:21.333 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:21.333 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:18:21.333 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:21.334 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:21.334 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:21.334 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:18:21.334 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:21.335 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:21.335 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:21.335 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:21.335 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:18:21.336 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:21.336 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:21.336 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:21.336 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:21.336 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:21.337 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:21.337 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:21.337 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:21.337 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:21.337 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:21.338 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:21.338 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:18:21.338 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:21.338 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:21.339 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:21.339 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:21.339 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:21.339 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:21.339 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:21.340 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:21.340 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:21.340 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:21.340 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:18:21.341 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:21.341 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:18:21.341 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:18:21.341 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:18:21.341 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:21.342 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:18:21.343 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:18:26.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:26.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:26.346 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:18:26.389 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:18:26.389 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:18:26.508 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:18:26.508 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:18:26.509 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:18:26.509 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:18:26.509 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:18:26.509 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:18:26.510 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:26.510 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:26.510 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:18:26.510 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:26.511 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:18:26.511 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:26.511 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:26.511 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:18:26.511 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:26.512 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:18:26.512 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:26.512 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:18:26.512 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:26.513 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:26.513 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:26.513 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:26.513 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:18:26.514 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:26.514 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:26.514 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:18:26.514 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:26.514 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:26.515 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:26.515 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:18:26.515 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:26.515 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:26.516 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:26.516 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:26.516 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:18:26.516 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:26.517 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:26.517 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:26.517 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:26.518 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:26.518 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:26.518 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:26.518 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:26.519 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:26.519 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:26.519 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:26.519 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:18:26.519 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:26.520 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:26.520 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:26.520 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:26.520 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:26.521 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:26.521 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:26.521 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:26.521 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:26.521 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:26.522 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:18:26.522 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:26.522 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:18:26.522 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:18:26.523 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:18:26.523 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:26.523 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:18:26.524 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:18:31.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:31.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:31.527 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:18:31.576 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:18:31.576 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:18:31.764 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:18:31.764 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:18:31.764 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:18:31.765 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:18:31.765 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:18:31.765 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:18:31.765 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:31.766 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:31.766 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:18:31.766 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:31.766 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:18:31.767 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:31.767 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:31.767 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:18:31.767 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:31.767 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:18:31.768 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:31.768 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:18:31.768 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:31.768 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:31.769 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:31.769 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:31.769 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:18:31.769 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:31.770 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:31.770 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:18:31.770 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:31.770 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:31.770 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:31.771 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:18:31.771 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:31.771 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:31.771 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:31.772 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:31.772 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:18:31.772 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:31.772 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:31.772 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:31.773 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:31.773 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:31.773 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:31.773 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:31.774 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:31.774 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:31.774 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:31.774 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:31.775 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:18:31.775 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:31.775 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:31.775 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:31.775 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:31.776 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:31.776 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:31.776 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:31.776 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:31.777 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:31.777 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:31.777 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:18:31.777 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:31.777 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:18:31.778 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:18:31.778 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:18:31.778 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:31.778 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:18:31.780 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:18:36.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:36.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:36.783 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:18:36.864 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:18:36.864 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:18:37.219 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:18:37.220 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:18:37.220 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:18:37.220 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:18:37.221 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:18:37.221 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:18:37.221 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:37.221 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:37.221 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:18:37.222 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:37.222 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:18:37.222 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:37.222 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:37.223 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:18:37.223 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:37.223 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:18:37.223 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:37.224 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:18:37.224 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:37.224 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:37.224 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:37.224 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:37.225 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:18:37.225 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:37.225 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:37.225 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:18:37.226 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:37.226 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:37.226 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:37.226 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:18:37.227 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:37.227 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:37.227 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:37.227 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:37.227 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:18:37.228 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:37.228 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:37.228 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:37.228 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:37.229 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:37.229 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:37.229 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:37.229 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:37.229 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:37.230 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:37.230 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:37.230 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:18:37.230 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:37.231 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:37.231 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:37.231 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:37.231 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:37.232 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:37.232 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:37.232 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:37.232 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:37.232 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:37.233 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:18:37.233 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:37.233 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:18:37.233 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:18:37.234 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:18:37.234 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:37.234 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:18:37.234 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.922243_318.1 2021-09-17T10:18:42.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:42.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:42.237 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:18:42.281 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:18:42.282 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:18:42.404 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:18:42.404 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:18:42.405 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:18:42.405 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:18:42.405 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:18:42.405 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:18:42.406 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:42.406 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:42.406 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:18:42.406 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:42.407 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:18:42.407 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:42.407 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:42.407 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:18:42.408 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:42.408 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:18:42.408 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:42.408 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:18:42.409 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:42.409 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:42.409 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:42.409 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:42.410 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:18:42.410 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:42.410 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:42.410 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:18:42.410 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:42.411 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:42.411 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:42.411 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:18:42.411 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:42.412 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:42.412 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:42.412 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:42.412 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:18:42.413 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:42.413 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:42.413 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:42.413 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:42.414 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:42.414 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:42.414 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:42.414 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:42.415 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:42.415 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:42.415 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:42.415 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:18:42.415 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:42.416 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:42.416 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:42.416 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:42.416 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:42.417 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:42.417 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:42.417 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:42.417 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:42.418 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:42.418 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:18:42.418 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:42.418 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:18:42.419 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:18:42.419 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:18:42.419 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:42.419 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:18:42.421 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:18:47.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:47.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:47.424 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:18:47.466 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:18:47.466 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:18:47.587 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:18:47.588 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:18:47.588 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:18:47.588 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:18:47.588 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:18:47.589 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:18:47.589 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:47.589 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:47.589 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:18:47.590 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:47.590 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:18:47.590 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:47.590 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:47.591 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:18:47.591 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:47.591 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:18:47.591 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:47.592 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:18:47.592 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:47.592 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:47.592 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:47.592 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:47.593 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:18:47.593 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:47.593 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:47.593 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:18:47.593 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:47.594 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:47.594 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:47.594 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:18:47.594 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:47.595 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:47.595 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:47.595 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:47.595 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:18:47.595 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:47.596 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:47.596 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:47.596 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:47.596 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:47.597 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:47.597 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:47.597 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:47.597 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:47.597 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:47.598 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:47.598 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:18:47.598 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:47.598 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:47.599 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:47.599 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:47.599 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:47.599 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:47.600 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:47.600 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:47.600 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:47.600 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:47.600 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:18:47.601 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:47.601 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:18:47.601 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:18:47.601 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:18:47.601 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:47.602 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:18:47.603 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:18:52.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:52.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:52.606 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:18:52.667 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:18:52.668 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:18:52.872 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:18:52.872 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:18:52.872 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:18:52.873 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:18:52.873 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:18:52.875 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:18:52.875 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:52.875 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:52.875 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:18:52.876 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:52.876 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:18:52.876 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:52.876 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:52.876 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:18:52.877 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:52.877 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:18:52.877 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:52.878 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:18:52.878 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:52.878 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:52.878 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:52.879 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:52.879 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:18:52.879 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:52.879 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:52.879 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:18:52.880 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:52.880 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:52.880 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:52.880 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:18:52.881 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:52.881 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:52.881 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:52.881 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:52.882 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:18:52.882 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:52.882 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:52.882 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:52.883 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:52.883 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:52.883 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:52.883 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:52.884 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:52.884 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:52.884 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:52.884 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:52.884 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:18:52.885 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:52.885 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:52.885 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:52.885 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:52.885 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:52.886 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:52.886 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:52.886 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:52.886 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:52.887 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:52.887 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:18:52.887 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:52.887 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:18:52.887 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:18:52.888 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:18:52.888 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:52.888 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:18:52.888 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:18:57.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:57.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:18:57.891 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:18:57.943 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:18:57.943 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:18:58.303 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:18:58.303 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:18:58.303 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:18:58.304 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:18:58.304 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:18:58.304 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:18:58.304 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:58.305 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:58.305 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:18:58.305 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:58.305 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:18:58.305 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:58.306 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:58.306 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:18:58.306 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:58.306 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:18:58.307 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:58.307 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:18:58.307 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:58.307 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:58.307 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:58.308 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:58.308 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:18:58.308 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:58.308 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:58.309 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:18:58.309 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:58.309 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:58.309 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:58.310 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:18:58.310 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:58.310 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:58.310 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:58.311 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:58.311 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:18:58.311 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:58.311 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:58.312 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:58.312 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:58.312 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:58.312 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:58.313 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:58.313 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:58.313 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:58.313 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:58.313 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:18:58.314 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:18:58.314 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:18:58.314 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:18:58.314 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:18:58.315 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:18:58.315 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:18:58.315 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:18:58.315 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:18:58.315 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:18:58.316 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:58.316 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:58.316 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:18:58.316 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:18:58.317 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:18:58.317 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:18:58.317 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:18:58.318 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:18:58.318 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:18:58.319 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.952206_320.1 2021-09-17T10:19:03.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:03.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:03.322 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:03.377 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:03.377 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:03.832 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:19:03.832 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:19:03.833 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:19:03.833 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:19:03.833 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:19:03.833 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:19:03.834 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:03.834 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:03.834 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:19:03.834 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:03.834 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:19:03.835 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:03.835 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:03.835 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:03.835 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:03.836 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:19:03.836 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:03.836 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:19:03.836 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:03.836 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:03.837 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:03.837 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:03.837 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:19:03.837 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:03.838 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:03.838 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:19:03.838 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:03.838 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:03.839 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:03.839 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:19:03.839 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:03.839 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:03.840 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:03.840 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:03.840 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:19:03.841 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:03.841 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:03.841 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:03.841 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:03.841 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:03.842 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:03.842 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:03.842 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:03.842 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:03.843 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:03.843 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:03.843 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:19:03.843 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:03.844 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:03.844 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:03.844 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:03.844 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:03.844 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:03.845 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:03.845 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:03.845 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:03.845 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:03.846 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:19:03.846 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:03.846 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:19:03.846 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:19:03.846 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:19:03.847 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:03.847 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:19:03.848 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:19:08.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:08.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:08.851 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:08.891 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:08.891 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:09.011 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:19:09.012 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:19:09.012 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:19:09.012 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:19:09.013 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:19:09.013 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:19:09.013 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:09.013 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:09.014 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:19:09.014 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:09.014 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:19:09.014 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:09.014 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:09.015 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:09.015 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:09.015 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:19:09.015 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:09.016 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:19:09.016 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:09.016 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:09.016 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:09.017 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:09.017 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:19:09.017 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:09.017 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:09.017 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:19:09.018 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:09.018 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:09.018 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:09.018 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:19:09.019 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:09.019 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:09.019 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:09.019 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:09.019 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:19:09.020 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:09.020 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:09.020 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:09.020 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:09.021 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:09.021 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:09.021 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:09.021 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:09.021 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:09.022 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:09.022 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:09.022 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:19:09.022 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:09.023 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:09.023 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:09.023 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:09.023 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:09.024 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:09.024 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:09.024 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:09.024 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:09.025 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:09.025 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:19:09.025 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:09.025 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:19:09.026 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:19:09.026 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:19:09.026 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:09.026 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:19:09.028 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.946914_319.1 2021-09-17T10:19:14.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:14.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:14.030 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:14.072 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:14.072 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:14.189 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:19:14.189 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:19:14.189 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:19:14.189 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:19:14.190 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:19:14.190 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:19:14.190 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:14.190 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:14.191 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:19:14.191 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:14.191 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:19:14.191 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:14.192 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:14.192 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:14.192 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:14.192 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:19:14.192 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:14.193 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:19:14.193 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:14.193 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:14.193 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:14.194 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:14.194 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:19:14.194 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:14.194 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:14.195 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:19:14.195 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:14.195 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:14.195 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:14.195 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:19:14.196 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:14.196 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:14.196 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:14.196 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:14.197 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:19:14.197 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:14.197 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:14.197 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:14.197 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:14.198 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:14.198 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:14.198 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:14.198 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:14.199 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:14.199 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:14.199 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:14.199 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:19:14.199 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:14.200 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:14.200 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:14.200 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:14.200 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:14.201 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:14.201 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:14.201 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:14.201 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:14.201 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:14.202 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:19:14.202 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:14.202 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:19:14.202 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:19:14.203 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:19:14.203 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:14.203 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:19:14.204 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.946914_319.1 2021-09-17T10:19:19.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:19.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:19.207 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:19.258 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:19.258 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:19.365 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:19:19.366 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:19:19.366 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:19:19.366 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:19:19.367 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:19:19.367 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:19:19.367 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:19.367 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:19.368 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:19:19.368 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:19.368 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:19:19.368 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:19.369 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:19.369 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:19.369 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:19.369 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:19:19.369 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:19.370 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:19:19.370 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:19.370 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:19.370 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:19.371 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:19.371 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:19:19.371 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:19.371 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:19.371 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.157966_331.1", 2021-09-17T10:19:19.372 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:19.372 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:19.372 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:19.372 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.157966Z", 2021-09-17T10:19:19.373 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:19.373 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:19.373 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:19.374 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:19.374 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:19:19.374 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:19.374 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:19.375 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:19.375 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:19.375 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:19.375 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:19.376 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:19.376 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:19.376 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:19.376 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:19.376 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:19.377 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:19:19.377 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:19.377 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:19.377 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:19.378 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:19.378 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:19.378 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:19.378 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:19.379 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:19.379 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:19.379 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:19.379 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:19:19.379 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:19.380 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:19:19.380 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:19:19.380 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:19:19.380 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:19.381 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:19:19.383 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.157966_331.1 2021-09-17T10:19:24.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:24.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:24.386 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:24.445 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:24.446 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:24.728 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:19:24.729 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:19:24.729 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:19:24.729 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:19:24.730 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:19:24.730 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:19:24.730 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:24.730 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:24.730 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:19:24.731 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:24.731 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:19:24.731 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:24.731 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:24.732 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:24.732 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:24.732 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:19:24.732 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:24.733 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:19:24.733 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:24.733 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:24.733 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:24.733 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:24.734 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:19:24.734 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:24.734 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:24.734 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:19:24.735 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:24.735 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:24.735 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:24.735 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:19:24.735 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:24.736 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:24.736 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:24.736 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:24.736 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:19:24.737 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:24.737 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:24.737 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:24.737 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:24.738 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:24.738 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:24.738 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:24.738 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:24.738 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:24.739 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:24.739 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:24.739 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:19:24.739 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:24.740 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:24.740 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:24.740 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:24.740 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:24.740 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:24.741 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:24.741 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:24.741 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:24.741 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:24.742 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:19:24.742 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:24.742 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:19:24.742 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:19:24.742 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:19:24.743 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:24.743 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:19:24.744 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.954369_321.1 2021-09-17T10:19:29.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:29.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:29.747 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:29.823 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:29.824 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:30.248 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:19:30.248 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:19:30.248 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:19:30.249 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:19:30.249 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:19:30.249 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:19:30.249 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:30.250 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:30.250 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:19:30.250 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:30.250 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:19:30.250 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:30.251 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:30.251 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:30.251 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:30.251 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:19:30.252 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:30.252 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:19:30.252 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:30.252 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:30.253 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:30.253 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:30.253 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:19:30.253 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:30.254 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:30.254 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.014731_326.1", 2021-09-17T10:19:30.254 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:30.254 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:30.254 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:30.255 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.014731Z", 2021-09-17T10:19:30.255 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:30.255 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:30.255 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:30.256 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:30.256 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:19:30.256 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:30.256 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:30.256 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:30.257 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:30.257 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:30.257 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:30.257 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:30.258 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:30.258 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:30.258 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:30.258 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:30.259 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:19:30.259 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:30.259 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:30.259 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:30.259 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:30.260 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:30.260 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:30.260 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:30.260 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:30.261 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:30.261 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:30.261 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:19:30.261 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:30.261 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:19:30.262 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:19:30.262 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:19:30.262 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:30.262 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:19:30.264 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.014731_326.1 2021-09-17T10:19:35.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:35.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:35.268 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:35.332 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:35.332 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:35.797 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:19:35.798 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:19:35.798 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:19:35.798 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:19:35.798 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:19:35.799 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:19:35.799 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:35.799 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:35.799 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:19:35.799 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:35.800 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:19:35.800 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:35.800 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:35.800 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:35.801 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:35.801 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:19:35.801 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:35.801 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:19:35.801 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:35.802 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:35.802 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:35.802 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:35.802 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:19:35.803 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:35.803 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:35.803 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.570872_349.1", 2021-09-17T10:19:35.803 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:35.803 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:35.804 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:35.804 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.570872Z", 2021-09-17T10:19:35.804 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:35.804 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:35.805 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:35.806 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:35.806 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:19:35.806 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:35.806 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:35.807 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:35.807 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:35.807 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:35.808 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:35.808 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:35.808 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:35.808 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:35.808 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:35.809 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:35.809 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:19:35.809 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:35.809 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:35.810 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:35.810 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:35.810 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:35.810 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:35.810 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:35.811 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:35.811 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:35.811 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:35.811 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:19:35.812 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:35.812 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:19:35.812 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:19:35.812 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:19:35.813 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:35.813 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:19:35.814 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873653.570872_349.1 2021-09-17T10:19:40.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:40.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:40.821 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:40.903 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:40.903 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:41.378 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:19:41.379 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:19:41.379 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:19:41.379 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:19:41.379 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:19:41.380 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:19:41.380 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:41.380 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:41.380 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:19:41.381 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:41.381 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:19:41.381 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:41.381 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:41.382 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:41.382 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:41.382 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:19:41.382 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:41.383 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:19:41.383 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:41.383 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:41.383 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:41.384 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:41.384 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:19:41.384 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:41.384 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:41.385 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:19:41.385 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:41.385 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:41.385 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:41.385 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:19:41.386 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:41.386 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:41.386 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:41.386 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:41.387 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:19:41.387 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:41.387 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:41.387 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:41.388 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:41.388 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:41.388 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:41.388 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:41.388 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:41.389 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:41.389 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:41.389 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:41.389 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:19:41.390 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:41.390 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:41.390 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:41.390 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:41.390 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:41.391 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:41.391 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:41.391 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:41.391 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:41.392 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:41.392 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:19:41.392 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:41.392 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:19:41.392 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:19:41.393 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:19:41.393 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:41.393 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:19:41.394 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873656.806856_380.1 target=1_1631873652.946914_319.1 2021-09-17T10:19:46.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2021-09-17T10:19:46.397 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-39 2021-09-17T10:19:46.440 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-40 2021-09-17T10:19:46.589 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-41 2021-09-17T10:19:46.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:46.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:46.978 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:47.068 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:47.069 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:47.251 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:19:47.281 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:19:47.281 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:47.282 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:47.282 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:19:47.282 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:47.282 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:47.283 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873986.809317_413.1", 2021-09-17T10:19:47.283 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:19:46.809317Z" 2021-09-17T10:19:47.283 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:47.283 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:47.284 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:47.284 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:19:47.284 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:47.284 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:47.284 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:47.285 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:19:47.285 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:47.285 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:19:47.286 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:19:47.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:47.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:47.288 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:47.331 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:47.332 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:47.444 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:19:47.444 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:19:47.444 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:19:47.444 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:19:47.445 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:19:47.445 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:19:47.445 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:19:47.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:19:47.446 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:19:47.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:19:47.446 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:19:47.446 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:19:47.447 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:19:47.447 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:47.447 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:19:47.447 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:19:47.447 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:19:47.448 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:19:47.448 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:19:47.448 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:19:47.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:19:47.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:19:47.449 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:19:47.449 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:19:47.449 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:19:47.450 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873986.440264_402.1", 2021-09-17T10:19:47.450 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:19:47.450 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:19:47.450 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:19:47.451 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:19:46.440264Z", 2021-09-17T10:19:47.451 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:19:47.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:19:47.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:19:47.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:19:47.452 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:19:47.452 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:19:47.453 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:19:47.453 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:19:47.453 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:19:47.453 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:19:47.453 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:19:47.454 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:19:47.454 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:19:47.454 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:19:47.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:19:47.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:19:47.455 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:19:47.455 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:19:47.455 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:19:47.456 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:19:47.456 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:19:47.456 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:19:47.456 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:19:47.456 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:19:47.457 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:19:47.457 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:19:47.457 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:19:47.457 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:19:47.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:19:47.458 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:19:47.458 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:19:47.458 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:19:47.459 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:19:47.459 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:19:47.460 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873986.440264_402.1 2021-09-17T10:19:52.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:52.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:52.463 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:52.508 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:52.509 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:52.625 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:19:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:19:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:19:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:19:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:19:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:19:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:19:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:19:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:19:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:19:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:19:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:19:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:19:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:19:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:19:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:19:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:19:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:19:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:19:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:19:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:19:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:19:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:19:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:19:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873986.809317_413.1", 2021-09-17T10:19:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:19:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:19:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:19:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:19:46.809317Z", 2021-09-17T10:19:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:19:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:19:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:19:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:19:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:19:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:19:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:19:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:19:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:19:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:19:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:19:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:19:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:19:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:19:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:19:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:19:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:19:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:19:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:19:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:19:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:19:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:19:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:19:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:19:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:19:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:19:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:19:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:19:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:19:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:19:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:19:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:19:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:19:52.642 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:19:52.643 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:19:52.644 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:19:52.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:52.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:52.647 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:52.734 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:52.735 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:53.124 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:19:53.124 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:19:53.124 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:19:53.125 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:19:53.125 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:19:53.125 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:19:53.125 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:53.126 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:53.126 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:19:53.126 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:53.126 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:19:53.127 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:53.127 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:53.127 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:53.127 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:53.127 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:19:53.128 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:53.128 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:19:53.129 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:53.129 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:53.129 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:53.129 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:53.129 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:19:53.130 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:53.130 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:53.130 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.044111_328.1", 2021-09-17T10:19:53.130 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:53.131 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:53.131 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:53.131 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.044111Z", 2021-09-17T10:19:53.131 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:53.132 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:53.132 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:53.132 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:53.132 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:19:53.132 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:53.133 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:53.133 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:53.133 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:53.133 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:53.133 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:53.134 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:53.134 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:53.134 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:53.134 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:53.135 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:53.135 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:19:53.135 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:53.135 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:53.135 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:53.136 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:53.136 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:53.136 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:53.136 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:53.137 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:53.137 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:53.137 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:53.137 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:19:53.137 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:53.138 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:19:53.138 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:19:53.138 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:19:53.138 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:53.138 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:19:53.140 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.044111_328.1 2021-09-17T10:19:58.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:58.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:19:58.142 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:19:58.184 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:19:58.184 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:19:58.304 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:19:58.304 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:19:58.305 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:19:58.305 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:19:58.305 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:19:58.305 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:19:58.306 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:58.306 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:58.306 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:19:58.306 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:58.307 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:19:58.307 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:58.307 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:58.307 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:19:58.307 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:58.308 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:19:58.308 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:58.308 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:19:58.308 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:58.308 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:58.309 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:58.309 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:58.309 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:19:58.309 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:58.310 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:58.310 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:19:58.310 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:58.310 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:58.311 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:58.311 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:19:58.311 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:58.311 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:58.311 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:58.312 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:58.312 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:19:58.312 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:58.312 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:58.312 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:58.313 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:58.313 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:58.313 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:58.313 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:58.314 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:58.314 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:58.314 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:58.314 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:19:58.314 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:19:58.315 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:19:58.315 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:19:58.315 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:19:58.315 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:19:58.316 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:19:58.316 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:19:58.316 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:19:58.316 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:19:58.316 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:58.317 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:58.317 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:19:58.317 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:19:58.317 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:19:58.318 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:19:58.318 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:19:58.318 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:19:58.318 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:19:58.319 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.009924_325.1 2021-09-17T10:20:03.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:03.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:03.322 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:20:03.398 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:20:03.398 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:20:03.701 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:20:03.701 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:20:03.701 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:20:03.701 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:20:03.702 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:20:03.702 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:20:03.702 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:03.702 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:03.703 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:20:03.703 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:03.703 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:20:03.703 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:03.704 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:03.704 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:20:03.704 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:03.704 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:20:03.704 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:03.705 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:20:03.705 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:03.705 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:03.705 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:03.706 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:03.706 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:20:03.706 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:03.706 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:03.707 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:20:03.707 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:03.707 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:03.707 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:03.707 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:20:03.708 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:03.708 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:03.708 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:03.708 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:03.709 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:20:03.709 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:03.709 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:03.709 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:03.709 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:03.710 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:03.710 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:03.710 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:03.711 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:03.711 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:03.711 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:03.711 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:03.711 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:20:03.712 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:03.712 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:03.712 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:03.712 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:03.713 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:03.713 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:03.713 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:03.713 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:03.713 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:20:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:20:03.715 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:20:03.715 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:20:03.715 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:03.715 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:20:03.717 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.952206_320.1 2021-09-17T10:20:08.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:08.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:08.720 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:20:08.763 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:20:08.763 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:20:08.884 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:20:08.884 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:20:08.884 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:20:08.885 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:20:08.886 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:20:08.886 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:20:08.886 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:08.887 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:08.887 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:20:08.887 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:08.888 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:20:08.888 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:08.888 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:08.889 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:20:08.889 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:08.889 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:20:08.889 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:08.890 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:20:08.890 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:08.890 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:08.890 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:08.891 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:08.891 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:20:08.891 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:08.891 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:08.892 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:20:08.892 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:08.892 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:08.892 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:08.893 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:20:08.893 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:08.893 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:08.893 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:08.893 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:08.894 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:20:08.894 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:08.894 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:08.894 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:08.895 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:08.895 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:08.895 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:08.895 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:08.896 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:08.896 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:08.896 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:08.896 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:08.896 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:20:08.897 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:08.897 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:08.897 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:08.897 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:08.898 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:08.898 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:08.898 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:08.899 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:08.899 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:08.899 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:08.900 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:20:08.900 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:08.900 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:20:08.900 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:20:08.901 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:20:08.901 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:08.901 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:20:08.903 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:20:13.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:13.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:13.906 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:20:13.954 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:20:13.954 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:20:14.046 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:20:14.047 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:20:14.047 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:20:14.047 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:20:14.047 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:20:14.048 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:20:14.048 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:14.048 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:14.048 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:20:14.049 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:14.049 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:20:14.049 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:14.049 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:14.050 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:20:14.050 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:14.050 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:20:14.050 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:14.051 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:20:14.051 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:14.051 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:14.051 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:14.051 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:14.052 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:20:14.052 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:14.052 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:14.052 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.099168_329.1", 2021-09-17T10:20:14.053 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:14.053 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:14.053 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:14.053 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.099168Z", 2021-09-17T10:20:14.053 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:14.054 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:14.054 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:14.054 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:14.054 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:20:14.054 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:14.055 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:14.055 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:14.055 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:14.055 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:14.056 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:14.056 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:14.056 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:14.056 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:14.057 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:14.057 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:14.057 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:20:14.057 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:14.057 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:14.058 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:14.058 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:14.058 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:14.058 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:14.059 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:14.059 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:14.059 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:14.059 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:14.059 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:20:14.060 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:14.060 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:20:14.060 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:20:14.060 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:20:14.061 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:14.061 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:20:14.061 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.099168_329.1 2021-09-17T10:20:19.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:19.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:19.064 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:20:19.106 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:20:19.106 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:20:19.225 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:20:19.226 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:20:19.226 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:20:19.226 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:20:19.227 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:20:19.227 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:20:19.227 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:19.227 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:19.228 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:20:19.228 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:19.228 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:20:19.228 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:19.228 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:19.229 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:20:19.229 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:19.229 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:20:19.229 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:19.230 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:20:19.230 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:19.230 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:19.230 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:19.231 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:19.231 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:20:19.231 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:19.231 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:19.232 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:20:19.232 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:19.232 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:19.232 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:19.232 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:20:19.233 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:19.233 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:19.233 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:19.233 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:19.234 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:20:19.234 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:19.234 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:19.234 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:19.235 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:19.235 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:19.235 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:19.235 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:19.236 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:19.236 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:19.236 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:19.236 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:19.236 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:20:19.237 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:19.237 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:19.237 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:19.237 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:19.238 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:19.238 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:19.238 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:19.238 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:19.239 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:19.239 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:19.239 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:20:19.239 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:19.240 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:20:19.240 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:20:19.240 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:20:19.240 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:19.241 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:20:19.242 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.946914_319.1 2021-09-17T10:20:24.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:24.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:24.244 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:20:24.286 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:20:24.287 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:20:24.369 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:20:24.370 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:20:24.370 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:20:24.370 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:20:24.371 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:20:24.371 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:20:24.371 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:24.371 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:24.372 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:20:24.372 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:24.372 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:20:24.372 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:24.373 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:24.373 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:20:24.373 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:24.373 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:20:24.373 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:24.374 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:20:24.374 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:24.374 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:24.374 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:24.375 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:24.375 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:20:24.375 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:24.375 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:24.376 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:20:24.376 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:24.376 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:24.376 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:24.376 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:20:24.377 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:24.377 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:24.377 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:24.377 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:24.378 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:20:24.378 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:24.378 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:24.378 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:24.379 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:24.379 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:24.379 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:24.379 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:24.380 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:24.380 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:24.380 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:24.380 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:24.380 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:20:24.381 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:24.381 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:24.381 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:24.381 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:24.382 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:24.382 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:24.382 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:24.382 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:24.382 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:24.383 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:24.383 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:20:24.383 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:24.383 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:20:24.384 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:20:24.384 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:20:24.384 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:24.384 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:20:24.386 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:20:29.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:29.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:29.388 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:20:29.431 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:20:29.431 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:20:29.617 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:20:29.618 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:20:29.618 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:20:29.618 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:20:29.619 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:20:29.619 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:20:29.619 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:29.619 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:29.620 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:20:29.620 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:29.620 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:20:29.620 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:29.620 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:29.621 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:20:29.621 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:29.621 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:20:29.621 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:29.622 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:20:29.622 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:29.622 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:29.622 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:29.622 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:29.623 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:20:29.623 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:29.623 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:29.623 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:20:29.624 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:29.624 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:29.624 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:29.624 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:20:29.624 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:29.625 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:29.625 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:29.625 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:29.625 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:20:29.626 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:29.626 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:29.626 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:29.626 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:29.626 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:29.627 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:29.627 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:29.627 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:29.627 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:29.628 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:29.628 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:29.628 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:20:29.628 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:29.629 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:29.629 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:29.629 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:29.629 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:29.629 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:29.630 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:29.630 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:29.630 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:29.630 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:29.631 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:20:29.631 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:29.631 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:20:29.631 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:20:29.632 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:20:29.632 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:29.632 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:20:29.633 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.922243_318.1 2021-09-17T10:20:34.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:34.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:34.636 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:20:34.677 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:20:34.678 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:20:34.763 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:20:34.764 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:20:34.764 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:20:34.765 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:20:34.765 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:20:34.765 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:20:34.765 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:34.766 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:34.766 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:20:34.766 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:34.766 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:20:34.767 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:34.767 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:34.768 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:20:34.768 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:34.768 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:20:34.768 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:34.769 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:20:34.769 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:34.769 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:34.769 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:34.770 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:34.770 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:20:34.770 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:34.771 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:34.771 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:20:34.771 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:34.772 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:34.773 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:34.773 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:20:34.773 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:34.773 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:34.774 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:34.774 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:34.774 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:20:34.775 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:34.775 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:34.775 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:34.775 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:34.775 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:34.776 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:34.776 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:34.776 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:34.776 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:34.777 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:34.777 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:34.777 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:20:34.777 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:34.777 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:34.778 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:34.778 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:34.778 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:34.779 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:34.779 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:34.779 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:34.779 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:34.780 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:34.780 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:20:34.780 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:34.780 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:20:34.780 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:20:34.781 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:20:34.781 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:34.781 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:20:34.782 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.946914_319.1 2021-09-17T10:20:39.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:39.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:39.786 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:20:39.879 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:20:39.879 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:20:40.326 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:20:40.327 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:20:40.327 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:20:40.327 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:20:40.328 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:20:40.328 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:20:40.328 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:40.328 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:40.329 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:20:40.329 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:40.329 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:20:40.329 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:40.329 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:40.330 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:20:40.330 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:40.330 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:20:40.330 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:40.331 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:20:40.331 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:40.331 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:40.331 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:40.332 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:40.332 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:20:40.332 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:40.332 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:40.332 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:20:40.333 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:40.333 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:40.333 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:40.333 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:20:40.333 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:40.334 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:40.334 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:40.334 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:40.334 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:20:40.335 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:40.335 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:40.335 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:40.335 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:40.335 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:40.336 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:40.336 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:40.336 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:40.336 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:40.337 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:40.337 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:40.337 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:20:40.337 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:40.338 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:40.338 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:40.338 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:40.338 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:40.338 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:40.339 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:40.339 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:40.339 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:40.339 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:40.340 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:20:40.340 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:40.340 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:20:40.340 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:20:40.340 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:20:40.341 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:40.341 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:20:40.342 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:20:45.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:45.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:45.345 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:20:45.388 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:20:45.389 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:20:45.509 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:20:45.509 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:20:45.509 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:20:45.509 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:20:45.510 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:20:45.510 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:20:45.510 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:45.510 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:45.511 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:20:45.511 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:45.511 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:20:45.511 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:45.512 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:45.512 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:20:45.512 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:45.512 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:20:45.512 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:45.513 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:20:45.513 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:45.513 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:45.513 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:45.514 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:45.514 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:20:45.514 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:45.515 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:45.515 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:20:45.515 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:45.515 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:45.515 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:45.516 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:20:45.516 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:45.516 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:45.517 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:45.517 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:45.517 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:20:45.517 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:45.518 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:45.518 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:45.518 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:45.518 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:45.519 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:45.519 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:45.519 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:45.519 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:45.519 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:45.520 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:45.520 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:20:45.520 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:45.520 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:45.521 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:45.521 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:45.521 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:45.521 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:45.521 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:45.522 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:45.522 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:45.522 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:45.522 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:20:45.523 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:45.523 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:20:45.523 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:20:45.523 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:20:45.523 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:45.524 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:20:45.525 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:20:50.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:50.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:50.528 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:20:50.587 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:20:50.588 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:20:50.911 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:20:50.911 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:20:50.911 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:20:50.912 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:20:50.912 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:20:50.913 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:20:50.913 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:50.913 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:50.914 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:20:50.914 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:50.914 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:20:50.914 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:50.915 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:50.915 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:20:50.915 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:50.915 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:20:50.916 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:50.916 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:20:50.916 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:50.916 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:50.917 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:50.917 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:50.917 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:20:50.917 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:50.918 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:50.918 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:20:50.918 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:50.918 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:50.919 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:50.919 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:20:50.919 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:50.919 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:50.920 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:50.920 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:50.920 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:20:50.920 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:50.921 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:50.921 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:50.921 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:50.921 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:50.921 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:50.922 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:50.922 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:50.922 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:50.922 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:50.923 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:50.923 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:20:50.923 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:50.923 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:50.924 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:50.924 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:50.924 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:50.924 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:50.924 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:50.925 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:50.925 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:50.925 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:50.925 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:20:50.926 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:50.926 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:20:50.926 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:20:50.926 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:20:50.926 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:50.927 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:20:50.928 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:20:55.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:55.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:20:55.931 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:20:55.974 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:20:55.974 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:20:56.092 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:20:56.093 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:20:56.093 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:20:56.093 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:20:56.093 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:20:56.094 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:20:56.094 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:56.094 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:56.094 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:20:56.095 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:56.095 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:20:56.095 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:56.095 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:56.096 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:20:56.096 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:56.096 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:20:56.096 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:56.097 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:20:56.097 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:56.097 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:56.097 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:56.098 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:56.098 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:20:56.098 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:56.098 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:56.099 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.412054_342.1", 2021-09-17T10:20:56.099 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:56.099 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:56.099 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:56.099 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.412054Z", 2021-09-17T10:20:56.100 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:56.100 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:56.100 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:56.100 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:56.101 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:20:56.101 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:56.101 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:56.101 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:56.102 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:56.102 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:56.102 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:56.102 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:56.103 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:56.103 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:56.103 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:56.103 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:20:56.104 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:20:56.104 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:20:56.104 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:20:56.104 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:20:56.105 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:20:56.105 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:20:56.105 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:20:56.105 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:20:56.105 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:20:56.106 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:56.106 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:56.106 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:20:56.106 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:20:56.107 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:20:56.107 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:20:56.107 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:20:56.107 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:20:56.108 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:20:56.109 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.412054_342.1 2021-09-17T10:21:01.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:01.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:01.112 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:21:01.172 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:21:01.172 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:21:01.445 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:21:01.446 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:21:01.446 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:21:01.446 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:21:01.446 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:21:01.447 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:21:01.447 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:01.447 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:01.447 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:21:01.448 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:01.448 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:21:01.448 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:01.448 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:01.448 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:21:01.449 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:01.449 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:21:01.450 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:01.450 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:21:01.450 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:01.450 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:01.451 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:01.451 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:01.451 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:21:01.451 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:01.452 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:01.452 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:21:01.452 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:01.452 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:01.452 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:01.453 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:21:01.453 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:01.453 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:01.453 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:01.454 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:01.454 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:21:01.454 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:01.454 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:01.455 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:01.455 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:01.456 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:01.456 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:01.456 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:01.456 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:01.458 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:01.458 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:01.458 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:01.459 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:21:01.459 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:01.459 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:01.461 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:01.461 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:01.461 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:01.461 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:01.461 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:01.462 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:01.462 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:01.462 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:01.462 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:21:01.463 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:01.463 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:21:01.463 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:21:01.463 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:21:01.463 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:01.464 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:21:01.465 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:21:06.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:06.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:06.469 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:21:06.513 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:21:06.513 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:21:06.634 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:21:06.635 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:21:06.635 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:21:06.635 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:21:06.635 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:21:06.636 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:21:06.636 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:06.636 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:06.636 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:21:06.636 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:06.637 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:21:06.637 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:06.637 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:06.637 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:21:06.638 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:06.638 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:21:06.638 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:06.638 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:21:06.638 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:06.639 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:06.639 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:06.639 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:06.639 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:21:06.640 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:06.640 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:06.640 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:21:06.640 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:06.640 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:06.641 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:06.641 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:21:06.641 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:06.641 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:06.641 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:06.642 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:06.642 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:21:06.642 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:06.642 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:06.643 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:06.643 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:06.643 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:06.643 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:06.643 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:06.644 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:06.644 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:06.644 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:06.644 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:06.645 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:21:06.645 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:06.645 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:06.645 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:06.645 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:06.646 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:06.646 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:06.646 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:06.646 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:06.646 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:06.647 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:06.647 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:21:06.647 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:06.647 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:21:06.648 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:21:06.648 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:21:06.648 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:06.648 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:21:06.650 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:21:11.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:11.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:11.652 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:21:11.700 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:21:11.700 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:21:11.805 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:21:11.806 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:21:11.806 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:21:11.806 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:21:11.807 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:21:11.807 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:21:11.807 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:11.807 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:11.808 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:21:11.808 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:11.808 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:21:11.808 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:11.808 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:11.809 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:21:11.809 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:11.809 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:21:11.809 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:11.810 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:21:11.810 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:11.810 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:11.810 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:11.810 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:11.811 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:21:11.811 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:11.811 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:11.811 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:21:11.812 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:11.812 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:11.812 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:11.812 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:21:11.812 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:11.813 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:11.813 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:11.814 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:11.814 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:21:11.814 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:11.814 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:11.815 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:11.815 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:11.815 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:11.815 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:11.816 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:11.816 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:11.816 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:11.816 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:11.817 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:11.817 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:21:11.817 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:11.817 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:11.817 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:11.818 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:11.818 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:11.818 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:11.818 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:11.819 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:11.819 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:11.819 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:11.819 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:21:11.819 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:11.820 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:21:11.820 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:21:11.820 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:21:11.820 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:11.821 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:21:11.822 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:21:16.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:16.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:16.825 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:21:16.866 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:21:16.867 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:21:17.123 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:21:17.123 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:21:17.124 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:21:17.124 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:21:17.124 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:21:17.124 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:21:17.125 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:17.125 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:17.125 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:21:17.126 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:17.126 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:21:17.126 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:17.126 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:17.126 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:21:17.127 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:17.127 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:21:17.127 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:17.127 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:21:17.128 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:17.128 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:17.128 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:17.128 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:17.128 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:21:17.129 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:17.129 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:17.129 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.001301_323.1", 2021-09-17T10:21:17.129 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:17.130 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:17.130 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:17.130 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.001301Z", 2021-09-17T10:21:17.131 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:17.131 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:17.131 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:17.131 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:17.131 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:21:17.132 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:17.132 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:17.132 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:17.134 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:17.134 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:17.135 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:17.135 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:17.135 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:17.136 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:17.136 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:17.137 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:17.137 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:21:17.137 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:17.138 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:17.138 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:17.139 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:17.139 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:17.140 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:17.140 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:17.140 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:17.141 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:17.141 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:17.142 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:21:17.142 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:17.143 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:21:17.143 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:21:17.143 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:21:17.144 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:17.144 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:21:17.146 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.001301_323.1 2021-09-17T10:21:22.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:22.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:22.150 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:21:22.194 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:21:22.194 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:21:22.277 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:21:22.278 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:21:22.278 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:21:22.278 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:21:22.278 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:21:22.279 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:21:22.279 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:22.279 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:22.279 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:21:22.280 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:22.280 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:21:22.280 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:22.280 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:22.280 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:21:22.281 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:22.281 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:21:22.282 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:22.282 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:21:22.282 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:22.282 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:22.283 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:22.283 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:22.283 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:21:22.283 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:22.284 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:22.284 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.925654_356.1", 2021-09-17T10:21:22.284 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:22.284 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:22.285 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:22.285 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.925654Z", 2021-09-17T10:21:22.285 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:22.285 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:22.286 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:22.286 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:22.286 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:21:22.286 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:22.287 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:22.287 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:22.287 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:22.287 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:22.287 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:22.288 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:22.288 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:22.288 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:22.288 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:22.288 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:22.289 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:21:22.289 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:22.289 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:22.289 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:22.290 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:22.290 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:22.290 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:22.290 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:22.290 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:22.291 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:22.291 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:22.291 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:21:22.291 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:22.292 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:21:22.292 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:21:22.292 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:21:22.292 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:22.292 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:21:22.294 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.925654_356.1 2021-09-17T10:21:27.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:27.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:27.297 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:21:27.399 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:21:27.400 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:21:27.821 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:21:27.822 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:21:27.822 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:21:27.822 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:21:27.823 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:21:27.823 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:21:27.823 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:27.823 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:27.824 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:21:27.824 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:27.824 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:21:27.824 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:27.824 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:27.825 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:21:27.825 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:27.825 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:21:27.825 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:27.826 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:21:27.826 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:27.826 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:27.826 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:27.827 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:27.827 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:21:27.827 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:27.827 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:27.828 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:21:27.828 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:27.828 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:27.828 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:27.829 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:21:27.829 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:27.829 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:27.829 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:27.829 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:27.830 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:21:27.830 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:27.830 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:27.830 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:27.831 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:27.831 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:27.831 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:27.831 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:27.831 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:27.832 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:27.832 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:27.832 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:27.832 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:21:27.833 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:27.833 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:27.834 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:27.834 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:27.834 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:27.835 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:27.835 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:27.835 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:27.835 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:27.835 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:27.836 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:21:27.836 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:27.836 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:21:27.836 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:21:27.837 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:21:27.837 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:27.837 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:21:27.839 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:21:32.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:32.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:32.843 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:21:32.886 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:21:32.886 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:21:33.007 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:21:33.008 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:21:33.008 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:21:33.009 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:21:33.009 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:21:33.009 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:21:33.010 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:33.010 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:33.011 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:21:33.011 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:33.011 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:21:33.012 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:33.012 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:33.012 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:21:33.013 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:33.013 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:21:33.013 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:33.014 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:21:33.014 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:33.014 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:33.014 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:33.015 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:33.015 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:21:33.015 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:33.015 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:33.016 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:21:33.016 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:33.017 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:33.017 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:33.017 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:21:33.018 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:33.018 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:33.019 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:33.019 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:33.020 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:21:33.020 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:33.020 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:33.021 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:33.021 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:33.022 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:33.022 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:33.023 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:33.023 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:33.023 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:33.024 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:33.024 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:33.025 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:21:33.025 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:33.026 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:33.026 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:33.027 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:33.027 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:33.027 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:33.028 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:33.028 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:33.029 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:33.029 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:33.029 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:21:33.030 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:33.030 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:21:33.031 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:21:33.031 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:21:33.032 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:33.032 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:21:33.034 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:21:38.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:38.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:38.036 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:21:38.110 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:21:38.110 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:21:38.446 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:21:38.446 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:21:38.447 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:21:38.447 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:21:38.447 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:21:38.447 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:21:38.448 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:38.448 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:38.448 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:21:38.448 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:38.449 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:21:38.449 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:38.449 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:38.449 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:21:38.449 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:38.450 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:21:38.450 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:38.450 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:21:38.450 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:38.451 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:38.451 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:38.451 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:38.451 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:21:38.452 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:38.452 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:38.452 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.038405_327.1", 2021-09-17T10:21:38.452 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:38.452 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:38.453 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:38.453 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.038405Z", 2021-09-17T10:21:38.453 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:38.453 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:38.454 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:38.454 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:38.454 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:21:38.454 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:38.455 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:38.455 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:38.455 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:38.455 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:38.456 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:38.456 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:38.456 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:38.456 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:38.457 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:38.457 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:38.457 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:21:38.457 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:38.459 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:38.459 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:38.459 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:38.459 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:38.460 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:38.460 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:38.460 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:38.460 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:38.461 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:38.461 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:21:38.462 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:38.462 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:21:38.462 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:21:38.463 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:21:38.463 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:38.463 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:21:38.465 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.038405_327.1 2021-09-17T10:21:43.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:43.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:43.469 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:21:43.512 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:21:43.512 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:21:43.628 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:21:43.628 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:21:43.628 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:21:43.629 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:21:43.629 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:21:43.629 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:21:43.629 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:43.630 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:43.630 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:21:43.630 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:43.630 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:21:43.631 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:43.631 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:43.631 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:21:43.631 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:43.631 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:21:43.632 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:43.632 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:21:43.632 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:43.632 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:43.633 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:43.633 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:43.633 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:21:43.633 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:43.633 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:43.634 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:21:43.634 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:43.634 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:43.634 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:43.635 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:21:43.635 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:43.635 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:43.635 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:43.635 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:43.636 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:21:43.636 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:43.636 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:43.636 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:43.637 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:43.637 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:43.637 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:43.637 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:43.637 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:43.638 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:43.638 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:43.638 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:43.638 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:21:43.639 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:43.639 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:43.639 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:43.639 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:43.639 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:43.640 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:43.640 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:43.640 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:43.640 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:43.640 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:43.641 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:21:43.641 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:43.641 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:21:43.641 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:21:43.642 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:21:43.642 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:43.642 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:21:43.643 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.009924_325.1 2021-09-17T10:21:48.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:48.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:48.651 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:21:48.710 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:21:48.710 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:21:48.823 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:21:48.823 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:21:48.823 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:21:48.824 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:21:48.824 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:21:48.824 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:21:48.824 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:48.825 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:48.825 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:21:48.825 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:48.825 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:21:48.825 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:48.826 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:48.826 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:21:48.826 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:48.826 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:21:48.827 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:48.827 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:21:48.827 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:48.827 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:48.828 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:48.828 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:48.828 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:21:48.828 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:48.829 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:48.829 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:21:48.829 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:48.830 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:48.830 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:48.830 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:21:48.830 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:48.831 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:48.831 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:48.831 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:48.831 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:21:48.832 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:48.832 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:48.832 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:48.832 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:48.832 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:48.833 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:48.833 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:48.833 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:48.833 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:48.834 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:48.834 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:48.834 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:21:48.834 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:48.835 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:48.835 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:48.835 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:48.835 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:48.835 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:48.836 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:48.836 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:48.836 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:48.836 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:48.836 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:21:48.837 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:48.837 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:21:48.837 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:21:48.837 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:21:48.838 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:48.838 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:21:48.840 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:21:53.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:53.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:53.842 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:21:53.923 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:21:53.923 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:21:54.409 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:21:54.410 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:21:54.410 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:21:54.410 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:21:54.411 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:21:54.411 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:21:54.411 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:54.411 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:54.412 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:21:54.412 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:54.412 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:21:54.412 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:54.412 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:54.413 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:21:54.413 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:54.413 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:21:54.413 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:54.414 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:21:54.414 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:54.414 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:54.414 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:54.415 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:54.415 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:21:54.415 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:54.415 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:54.416 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:21:54.416 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:54.416 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:54.416 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:54.416 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:21:54.417 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:54.418 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:54.418 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:54.418 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:54.418 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:21:54.419 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:54.419 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:54.419 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:54.419 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:54.420 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:54.420 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:54.420 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:54.420 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:54.420 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:54.421 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:54.421 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:54.421 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:21:54.421 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:54.422 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:54.422 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:54.422 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:54.422 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:54.423 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:54.423 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:54.423 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:54.423 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:54.423 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:54.424 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:21:54.424 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:54.424 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:21:54.424 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:21:54.425 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:21:54.425 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:54.425 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:21:54.426 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:21:59.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:59.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:21:59.430 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:21:59.476 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:21:59.476 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:21:59.587 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:21:59.587 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:21:59.588 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:21:59.588 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:21:59.588 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:21:59.588 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:21:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:21:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:21:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:21:59.591 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:59.591 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:21:59.591 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:59.591 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:21:59.591 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:59.592 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:59.592 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:59.592 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:59.592 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:21:59.593 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:59.593 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:59.593 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873654.157940_361.1", 2021-09-17T10:21:59.594 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:59.594 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:59.594 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:59.595 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:14.157940Z", 2021-09-17T10:21:59.595 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:59.595 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:59.595 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:59.596 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:59.596 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:21:59.596 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:59.596 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:59.597 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:59.597 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:59.597 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:59.597 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:59.597 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:59.598 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:59.598 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:59.598 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:59.598 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:21:59.599 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:21:59.599 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:21:59.599 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:21:59.599 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:21:59.599 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:21:59.600 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:21:59.600 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:21:59.600 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:21:59.600 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:21:59.600 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:59.601 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:59.601 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:21:59.601 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:21:59.601 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:21:59.602 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:21:59.602 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:21:59.602 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:21:59.602 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:21:59.603 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873654.157940_361.1 2021-09-17T10:22:04.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:04.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:04.606 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:22:04.691 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:22:04.692 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:22:04.989 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:22:04.989 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:22:04.989 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:22:04.990 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:22:04.990 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:22:04.990 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:22:04.990 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:04.991 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:04.991 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:22:04.991 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:04.991 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:22:04.992 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:04.992 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:04.992 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:22:04.992 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:04.992 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:22:04.993 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:04.993 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:22:04.993 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:04.993 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:04.994 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:04.994 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:04.994 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:22:04.994 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:04.995 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:04.995 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:22:04.995 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:04.995 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:04.995 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:04.996 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:22:04.996 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:04.996 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:04.996 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:04.997 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:04.998 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:22:04.998 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:04.998 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:04.998 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:04.999 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:04.999 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:04.999 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:04.999 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:05.000 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:05.000 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:05.000 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:05.000 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:05.000 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:22:05.001 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:05.001 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:05.001 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:05.001 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:05.002 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:05.002 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:05.002 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:05.002 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:05.002 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:05.003 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:05.003 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:22:05.003 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:05.003 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:22:05.004 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:22:05.004 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:22:05.004 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:05.004 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:22:05.006 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.946914_319.1 2021-09-17T10:22:10.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:10.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:10.009 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:22:10.051 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:22:10.051 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:22:10.172 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:22:10.173 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:22:10.173 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:22:10.173 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:22:10.173 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:22:10.173 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:22:10.174 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:10.174 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:10.174 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:22:10.174 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:10.175 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:22:10.175 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:10.175 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:10.175 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:22:10.176 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:10.176 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:22:10.176 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:10.176 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:22:10.176 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:10.177 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:10.177 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:10.177 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:10.177 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:22:10.177 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:10.178 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:10.178 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:22:10.178 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:10.178 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:10.179 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:10.179 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:22:10.179 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:10.179 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:10.179 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:10.180 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:10.180 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:22:10.180 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:10.180 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:10.181 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:10.181 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:10.181 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:10.181 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:10.181 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:10.182 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:10.182 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:10.182 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:10.182 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:10.183 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:22:10.183 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:10.183 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:10.183 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:10.183 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:10.184 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:10.184 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:10.184 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:10.185 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:10.185 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:10.185 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:10.185 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:22:10.185 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:10.186 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:22:10.186 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:22:10.186 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:22:10.186 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:10.187 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:22:10.188 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.952206_320.1 2021-09-17T10:22:15.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:15.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:15.191 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:22:15.238 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:22:15.239 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:22:15.348 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:22:15.348 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:22:15.349 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:22:15.349 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:22:15.349 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:22:15.350 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:22:15.350 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:15.351 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:15.351 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:22:15.352 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:15.352 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:22:15.352 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:15.353 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:15.353 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:22:15.353 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:15.353 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:22:15.354 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:15.354 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:22:15.355 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:15.358 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:15.359 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:15.359 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:15.359 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:22:15.359 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:15.360 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:15.360 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:22:15.360 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:15.360 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:15.360 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:15.361 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:22:15.361 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:15.361 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:15.362 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:15.362 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:15.363 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:22:15.363 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:15.363 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:15.363 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:15.364 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:15.364 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:15.364 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:15.364 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:15.365 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:15.365 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:15.365 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:15.365 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:15.365 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:22:15.366 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:15.366 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:15.366 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:15.366 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:15.367 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:15.367 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:15.367 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:15.368 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:15.368 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:15.368 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:15.368 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:22:15.368 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:15.369 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:22:15.369 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:22:15.369 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:22:15.369 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:15.370 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:22:15.371 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:22:20.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:20.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:20.373 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:22:20.444 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:22:20.445 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:22:20.823 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:22:20.823 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:22:20.823 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:22:20.823 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:22:20.824 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:22:20.824 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:22:20.824 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:20.824 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:20.825 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:22:20.825 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:20.825 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:22:20.825 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:20.826 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:20.826 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:22:20.826 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:20.826 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:22:20.827 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:20.827 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:22:20.827 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:20.827 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:20.827 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:20.828 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:20.828 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:22:20.828 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:20.828 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:20.829 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:22:20.829 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:20.829 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:20.829 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:20.830 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:22:20.830 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:20.830 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:20.830 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:20.831 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:20.831 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:22:20.831 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:20.831 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:20.831 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:20.832 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:20.832 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:20.832 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:20.832 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:20.833 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:20.833 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:20.833 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:20.833 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:20.833 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:22:20.834 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:20.834 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:20.834 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:20.834 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:20.835 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:20.835 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:20.835 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:20.835 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:20.835 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:20.836 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:20.836 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:22:20.836 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:20.836 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:22:20.837 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:22:20.837 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:22:20.837 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:20.837 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:22:20.839 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.946914_319.1 2021-09-17T10:22:25.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:25.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:25.841 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:22:25.882 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:22:25.883 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:22:26.003 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:22:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:22:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:22:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:22:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:22:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:22:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:22:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:22:26.007 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:26.008 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:26.008 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:22:26.008 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:26.008 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:22:26.009 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:26.009 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:22:26.009 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:26.009 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:26.010 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:26.010 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:26.010 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:22:26.010 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:26.010 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:26.011 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.959198_322.1", 2021-09-17T10:22:26.011 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:26.011 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:26.011 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:26.012 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.959198Z", 2021-09-17T10:22:26.012 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:26.012 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:26.012 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:26.013 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:26.013 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:22:26.013 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:26.013 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:26.014 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:26.014 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:26.014 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:26.014 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:26.015 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:26.015 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:26.015 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:26.016 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:26.016 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:26.016 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:22:26.016 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:26.017 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:26.017 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:26.018 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:26.018 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:26.018 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:26.018 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:26.019 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:26.019 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:26.019 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:26.019 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:22:26.020 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:26.020 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:22:26.020 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:22:26.020 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:22:26.020 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:26.021 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:22:26.022 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.959198_322.1 2021-09-17T10:22:31.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:31.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:31.025 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:22:31.099 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:22:31.099 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:22:31.322 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:22:31.323 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:22:31.323 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:22:31.323 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:22:31.323 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:22:31.324 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:22:31.324 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:31.324 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:31.324 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:22:31.325 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:31.325 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:22:31.325 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:31.325 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:31.325 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:22:31.326 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:31.326 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:22:31.326 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:31.326 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:22:31.327 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:31.327 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:31.327 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:31.327 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:31.327 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:22:31.328 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:31.328 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:31.328 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:22:31.328 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:31.329 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:31.329 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:31.329 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:22:31.329 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:31.329 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:31.330 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:31.330 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:31.331 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:22:31.331 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:31.331 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:31.331 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:31.332 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:31.332 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:31.332 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:31.332 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:31.332 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:31.333 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:31.333 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:31.333 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:31.333 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:22:31.333 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:31.334 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:31.334 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:31.334 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:31.334 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:31.335 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:31.335 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:31.335 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:31.335 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:31.335 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:31.336 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:22:31.336 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:31.336 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:22:31.336 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:22:31.337 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:22:31.337 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:31.337 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:22:31.338 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:22:36.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:36.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:36.341 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:22:36.569 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:22:36.570 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:22:36.928 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:22:36.929 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:22:36.929 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:22:36.929 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:22:36.930 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:22:36.930 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:22:36.930 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:36.931 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:36.931 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:22:36.931 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:36.932 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:22:36.932 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:36.932 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:36.932 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:22:36.933 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:36.933 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:22:36.933 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:36.933 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:22:36.934 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:36.934 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:36.934 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:36.935 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:36.935 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:22:36.935 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:36.935 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:36.936 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:22:36.936 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:36.936 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:36.936 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:36.937 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:22:36.937 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:36.937 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:36.937 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:36.937 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:36.938 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:22:36.938 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:36.938 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:36.938 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:36.939 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:36.939 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:36.939 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:36.939 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:36.939 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:36.940 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:36.940 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:36.940 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:36.940 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:22:36.941 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:36.941 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:36.941 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:36.942 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:36.942 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:36.943 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:36.943 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:36.943 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:36.945 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:36.945 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:36.945 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:22:36.946 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:36.946 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:22:36.946 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:22:36.947 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:22:36.947 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:36.947 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:22:36.948 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.952206_320.1 2021-09-17T10:22:41.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:41.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:41.951 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:22:42.050 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:22:42.050 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:22:42.172 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:22:42.173 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:22:42.173 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:22:42.173 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:22:42.174 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:22:42.174 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:22:42.174 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:42.174 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:42.175 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:22:42.175 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:42.175 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:22:42.175 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:42.176 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:42.176 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:22:42.176 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:42.176 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:22:42.177 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:42.177 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:22:42.177 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:42.177 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:42.177 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:42.178 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:42.178 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:22:42.178 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:42.178 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:42.179 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:22:42.179 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:42.179 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:42.179 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:42.180 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:22:42.180 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:42.180 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:42.180 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:42.181 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:42.181 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:22:42.181 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:42.182 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:42.182 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:42.183 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:42.183 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:42.183 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:42.184 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:42.184 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:42.184 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:42.185 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:42.185 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:42.185 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:22:42.186 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:42.186 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:42.187 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:42.187 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:42.187 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:42.187 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:42.188 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:42.188 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:42.188 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:42.188 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:42.189 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:22:42.189 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:42.189 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:22:42.189 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:22:42.190 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:22:42.190 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:42.190 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:22:42.191 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:22:47.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:47.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:47.195 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:22:47.338 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:22:47.338 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:22:47.609 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:22:47.610 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:22:47.610 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:22:47.610 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:22:47.611 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:22:47.611 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:22:47.611 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:47.612 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:47.612 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:22:47.613 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:47.613 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:22:47.613 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:47.613 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:47.613 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:22:47.614 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:47.615 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:22:47.615 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:47.615 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:22:47.615 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:47.616 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:47.616 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:47.616 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:47.616 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:22:47.617 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:47.617 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:47.617 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:22:47.617 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:47.618 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:47.618 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:47.618 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:22:47.618 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:47.618 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:47.619 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:47.619 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:47.619 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:22:47.620 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:47.620 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:47.620 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:47.620 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:47.620 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:47.621 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:47.621 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:47.621 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:47.622 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:47.622 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:47.622 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:47.623 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:22:47.623 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:47.623 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:47.623 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:47.624 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:47.624 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:47.624 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:47.624 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:47.625 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:47.625 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:47.626 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:47.626 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:22:47.626 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:47.626 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:22:47.626 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:22:47.627 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:22:47.627 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:47.627 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:22:47.628 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:22:52.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:52.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:52.631 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:22:53.059 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:22:53.059 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:22:53.489 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:22:53.490 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:22:53.490 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:22:53.490 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:22:53.491 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:22:53.491 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:22:53.491 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:53.491 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:53.492 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:22:53.492 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:53.492 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:22:53.492 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:53.493 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:53.493 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:22:53.493 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:53.493 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:22:53.493 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:53.494 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:22:53.494 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:53.494 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:53.494 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:53.495 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:53.495 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:22:53.495 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:53.495 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:53.496 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:22:53.496 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:53.496 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:53.496 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:53.496 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:22:53.497 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:53.497 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:53.497 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:53.497 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:53.498 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:22:53.498 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:53.498 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:53.498 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:53.499 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:53.499 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:53.499 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:53.499 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:53.499 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:53.500 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:53.500 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:53.500 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:53.500 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:22:53.501 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:53.501 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:53.501 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:53.501 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:53.501 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:53.502 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:53.502 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:53.502 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:53.502 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:53.503 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:53.503 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:22:53.503 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:53.503 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:22:53.503 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:22:53.504 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:22:53.504 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:53.504 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:22:53.505 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:22:58.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:58.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:22:58.507 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:22:58.551 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:22:58.551 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:22:58.669 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:22:58.669 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:22:58.669 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:22:58.670 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:22:58.670 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:22:58.670 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:22:58.670 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:58.671 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:58.671 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:22:58.671 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:58.671 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:22:58.671 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:58.672 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:58.672 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:22:58.672 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:58.672 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:22:58.673 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:58.673 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:22:58.673 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:58.673 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:58.673 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:58.674 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:58.674 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:22:58.674 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:58.674 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:58.675 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.344112_340.1", 2021-09-17T10:22:58.675 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:58.675 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:58.675 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:58.676 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.344112Z", 2021-09-17T10:22:58.676 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:58.676 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:58.676 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:58.676 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:58.677 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:22:58.677 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:58.677 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:58.677 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:58.678 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:58.678 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:58.678 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:58.678 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:58.679 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:58.679 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:58.679 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:58.679 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:22:58.679 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:22:58.680 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:22:58.680 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:22:58.680 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:22:58.680 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:22:58.681 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:22:58.681 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:22:58.681 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:22:58.681 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:22:58.682 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:58.682 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:58.682 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:22:58.682 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:22:58.683 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:22:58.683 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:22:58.683 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:22:58.683 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:22:58.683 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:22:58.685 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.344112_340.1 2021-09-17T10:23:03.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:03.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:03.688 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:23:03.773 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:23:03.774 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:23:04.103 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:23:04.103 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:23:04.104 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:23:04.104 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:23:04.104 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:23:04.104 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:23:04.105 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:04.105 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:04.105 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:23:04.105 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:04.105 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:23:04.106 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:04.106 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:04.106 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:23:04.106 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:04.107 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:23:04.107 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:04.107 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:23:04.107 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:04.107 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:04.108 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:04.108 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:04.108 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:23:04.108 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:04.109 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:04.109 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.014731_326.1", 2021-09-17T10:23:04.109 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:04.109 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:04.109 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:04.110 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.014731Z", 2021-09-17T10:23:04.110 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:04.110 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:04.110 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:04.111 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:04.111 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:23:04.111 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:04.111 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:04.112 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:04.112 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:04.112 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:04.113 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:04.113 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:04.113 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:04.113 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:04.114 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:04.114 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:04.114 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:23:04.114 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:04.115 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:04.115 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:04.115 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:04.115 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:04.116 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:04.116 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:04.116 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:04.116 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:04.116 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:04.117 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:23:04.117 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:04.117 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:23:04.117 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:23:04.118 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:23:04.118 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:04.118 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:23:04.119 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.014731_326.1 2021-09-17T10:23:09.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:09.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:09.122 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:23:09.181 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:23:09.182 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:23:09.302 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:23:09.302 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:23:09.303 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:23:09.303 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:23:09.303 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:23:09.304 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:23:09.304 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:09.305 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:09.305 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:23:09.306 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:09.306 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:23:09.307 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:09.307 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:09.307 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:23:09.308 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:09.308 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:23:09.308 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:09.309 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:23:09.310 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:09.310 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:09.310 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:09.310 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:09.311 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:23:09.311 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:09.311 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:09.311 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.324538_337.1", 2021-09-17T10:23:09.311 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:09.312 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:09.312 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:09.312 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.324538Z", 2021-09-17T10:23:09.312 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:09.313 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:09.313 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:09.313 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:09.313 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:23:09.313 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:09.314 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:09.314 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:09.314 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:09.314 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:09.315 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:09.315 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:09.315 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:09.316 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:09.316 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:09.316 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:09.316 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:23:09.316 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:09.317 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:09.318 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:09.318 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:09.319 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:09.319 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:09.319 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:09.319 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:09.320 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:09.320 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:09.320 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:23:09.320 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:09.320 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:23:09.321 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:23:09.321 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:23:09.321 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:09.321 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:23:09.323 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.324538_337.1 2021-09-17T10:23:14.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:14.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:14.326 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:23:14.421 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:23:14.421 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:23:14.559 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:23:14.559 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:23:14.559 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:23:14.560 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:23:14.560 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:23:14.560 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:23:14.560 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:14.561 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:14.561 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:23:14.561 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:14.561 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:23:14.562 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:14.562 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:14.562 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:23:14.562 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:14.562 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:23:14.563 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:14.563 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:23:14.563 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:14.563 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:14.564 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:14.564 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:14.564 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:23:14.564 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:14.564 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:14.565 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:23:14.565 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:14.565 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:14.565 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:14.566 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:23:14.566 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:14.566 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:14.566 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:14.566 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:14.567 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:23:14.567 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:14.567 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:14.567 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:14.568 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:14.568 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:14.568 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:14.568 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:14.569 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:14.569 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:14.569 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:14.569 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:14.570 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:23:14.570 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:14.570 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:14.570 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:14.570 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:14.571 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:14.571 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:14.571 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:14.571 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:14.572 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:14.572 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:14.572 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:23:14.572 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:14.573 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:23:14.573 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:23:14.573 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:23:14.573 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:14.574 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:23:14.575 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:23:19.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:19.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:19.577 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:23:19.620 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:23:19.620 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:23:19.742 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:23:19.743 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:23:19.743 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:23:19.743 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:23:19.744 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:23:19.744 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:23:19.744 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:19.744 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:19.744 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:23:19.745 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:19.745 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:23:19.745 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:19.745 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:19.746 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:23:19.746 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:19.746 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:23:19.746 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:19.747 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:23:19.747 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:19.747 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:19.747 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:19.747 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:19.748 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:23:19.748 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:19.748 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:19.748 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:23:19.749 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:19.749 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:19.749 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:19.749 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:23:19.749 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:19.750 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:19.750 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:19.751 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:19.751 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:23:19.751 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:19.751 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:19.752 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:19.752 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:19.752 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:19.752 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:19.752 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:19.753 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:19.753 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:19.753 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:19.753 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:19.754 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:23:19.755 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:19.756 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:19.756 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:19.756 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:19.756 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:19.756 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:19.757 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:19.757 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:19.757 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:19.757 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:19.758 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:23:19.758 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:19.758 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:23:19.758 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:23:19.759 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:23:19.759 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:19.759 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:23:19.760 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.009924_325.1 2021-09-17T10:23:24.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:24.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:24.763 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:23:24.823 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:23:24.823 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:23:24.924 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:23:24.924 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:23:24.925 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:23:24.925 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:23:24.925 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:23:24.925 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:23:24.925 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:24.926 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:24.926 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:23:24.926 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:24.926 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:23:24.927 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:24.927 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:24.927 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:23:24.927 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:24.928 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:23:24.928 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:24.928 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:23:24.928 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:24.929 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:24.929 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:24.929 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:24.929 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:23:24.930 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:24.930 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:24.930 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.925654_356.1", 2021-09-17T10:23:24.930 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:24.930 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:24.931 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:24.931 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.925654Z", 2021-09-17T10:23:24.931 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:24.931 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:24.932 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:24.932 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:24.932 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:23:24.932 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:24.932 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:24.933 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:24.933 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:24.933 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:24.933 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:24.933 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:24.934 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:24.934 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:24.934 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:24.934 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:24.935 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:23:24.935 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:24.935 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:24.935 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:24.936 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:24.936 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:24.936 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:24.936 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:24.936 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:24.937 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:24.937 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:24.937 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:23:24.937 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:24.938 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:23:24.938 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:23:24.938 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:23:24.938 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:24.938 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:23:24.940 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.925654_356.1 2021-09-17T10:23:29.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:29.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:29.943 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:23:30.069 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:23:30.069 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:23:30.520 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:23:30.520 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:23:30.521 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:23:30.521 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:23:30.521 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:23:30.521 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:23:30.522 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:30.522 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:30.522 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:23:30.522 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:30.523 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:23:30.523 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:30.523 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:30.523 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:23:30.524 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:30.524 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:23:30.524 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:30.524 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:23:30.524 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:30.525 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:30.525 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:30.525 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:30.525 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:23:30.526 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:30.526 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:30.526 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:23:30.526 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:30.527 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:30.527 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:30.527 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:23:30.527 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:30.528 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:30.528 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:30.528 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:30.528 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:23:30.528 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:30.529 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:30.529 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:30.529 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:30.529 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:30.530 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:30.530 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:30.530 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:30.530 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:30.531 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:30.531 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:30.531 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:23:30.532 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:30.532 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:30.532 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:30.532 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:30.533 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:30.533 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:30.533 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:30.533 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:30.533 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:30.534 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:30.534 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:23:30.534 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:30.534 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:23:30.535 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:23:30.535 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:23:30.535 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:30.535 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:23:30.537 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.009924_325.1 2021-09-17T10:23:35.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:35.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:35.540 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:23:35.583 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:23:35.583 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:23:35.709 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:23:35.709 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:23:35.710 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:23:35.710 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:23:35.710 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:23:35.710 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:23:35.711 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:35.711 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:35.711 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:23:35.711 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:35.711 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:23:35.712 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:35.712 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:35.712 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:23:35.713 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:35.713 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:23:35.713 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:35.713 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:23:35.713 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:35.714 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:35.714 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:35.714 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:35.715 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:23:35.715 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:35.715 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:35.715 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.044111_328.1", 2021-09-17T10:23:35.716 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:35.716 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:35.717 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:35.717 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.044111Z", 2021-09-17T10:23:35.717 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:35.717 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:35.718 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:35.718 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:35.718 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:23:35.718 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:35.719 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:35.719 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:35.719 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:35.720 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:35.720 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:35.720 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:35.720 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:35.720 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:35.721 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:35.721 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:35.721 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:23:35.721 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:35.722 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:35.722 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:35.722 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:35.722 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:35.723 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:35.723 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:35.723 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:35.723 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:35.724 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:35.724 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:23:35.724 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:35.725 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:23:35.725 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:23:35.725 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:23:35.725 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:35.726 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:23:35.727 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.044111_328.1 2021-09-17T10:23:40.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:40.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:40.732 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:23:40.834 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:23:40.835 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:23:41.069 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:23:41.070 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:23:41.070 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:23:41.070 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:23:41.070 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:23:41.071 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:23:41.071 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:41.071 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:41.071 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:23:41.072 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:41.072 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:23:41.072 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:41.072 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:41.072 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:23:41.073 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:41.073 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:23:41.073 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:41.073 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:23:41.074 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:41.074 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:41.074 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:41.074 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:41.075 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:23:41.075 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:41.075 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:41.075 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.336676_339.1", 2021-09-17T10:23:41.076 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:41.076 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:41.076 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:41.076 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.336676Z", 2021-09-17T10:23:41.076 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:41.077 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:41.077 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:41.077 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:41.077 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:23:41.078 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:41.078 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:41.078 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:41.078 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:41.078 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:41.079 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:41.079 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:41.079 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:41.079 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:41.080 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:41.080 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:41.080 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:23:41.080 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:41.080 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:41.081 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:41.081 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:41.081 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:41.081 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:41.082 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:41.082 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:41.082 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:41.082 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:41.082 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:23:41.083 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:41.083 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:23:41.083 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:23:41.083 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:23:41.084 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:41.084 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:23:41.085 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.336676_339.1 2021-09-17T10:23:46.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:46.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:46.088 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:23:46.133 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:23:46.133 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:23:46.251 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:23:46.251 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:23:46.252 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:23:46.252 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:23:46.252 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:23:46.252 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:23:46.253 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:46.253 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:46.253 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:23:46.254 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:46.254 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:23:46.254 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:46.255 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:46.255 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:23:46.255 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:46.256 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:23:46.256 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:46.256 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:23:46.257 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:46.257 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:46.258 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:46.258 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:46.258 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:23:46.259 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:46.259 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:46.260 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.336676_339.1", 2021-09-17T10:23:46.260 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:46.260 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:46.261 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:46.261 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.336676Z", 2021-09-17T10:23:46.261 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:46.262 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:46.262 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:46.262 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:46.263 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:23:46.263 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:46.264 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:46.264 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:46.264 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:46.265 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:46.265 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:46.265 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:46.265 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:46.266 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:46.266 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:46.266 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:46.266 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:23:46.267 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:46.267 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:46.267 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:46.267 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:46.268 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:46.268 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:46.268 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:46.268 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:46.269 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:46.269 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:46.269 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:23:46.269 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:46.269 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:23:46.270 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:23:46.270 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:23:46.270 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:46.270 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:23:46.272 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.336676_339.1 2021-09-17T10:23:51.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:51.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:51.275 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:23:51.316 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:23:51.316 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:23:51.405 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:23:51.405 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:23:51.406 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:23:51.406 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:23:51.406 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:23:51.406 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:23:51.407 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:51.407 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:51.407 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:23:51.408 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:51.408 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:23:51.408 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:51.408 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:51.409 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:23:51.409 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:51.409 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:23:51.409 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:51.410 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:23:51.410 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:51.410 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:51.410 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:51.411 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:51.411 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:23:51.411 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:51.411 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:51.411 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:23:51.412 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:51.412 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:51.412 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:51.412 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:23:51.412 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:51.413 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:51.413 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:51.413 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:51.413 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:23:51.414 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:51.414 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:51.414 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:51.414 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:51.415 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:51.415 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:51.415 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:51.415 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:51.416 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:51.416 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:51.416 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:51.416 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:23:51.416 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:51.417 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:51.417 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:51.417 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:51.417 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:51.417 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:51.418 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:51.418 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:51.418 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:51.418 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:51.419 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:23:51.419 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:51.419 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:23:51.419 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:23:51.420 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:23:51.420 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:51.420 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:23:51.421 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.946914_319.1 2021-09-17T10:23:56.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:56.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:23:56.424 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:23:56.565 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:23:56.565 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:23:56.754 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:23:56.755 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:23:56.755 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:23:56.755 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:23:56.755 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:23:56.756 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:23:56.756 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:56.756 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:56.756 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:23:56.757 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:56.757 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:23:56.757 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:56.757 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:56.757 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:23:56.758 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:56.758 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:23:56.758 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:56.758 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:23:56.759 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:56.759 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:56.759 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:56.759 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:56.760 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:23:56.760 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:56.760 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:56.760 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.336676_339.1", 2021-09-17T10:23:56.760 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:56.761 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:56.761 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:56.761 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.336676Z", 2021-09-17T10:23:56.761 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:56.762 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:56.762 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:56.762 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:56.762 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:23:56.762 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:56.763 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:56.763 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:56.763 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:56.763 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:56.764 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:56.764 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:56.764 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:56.764 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:56.764 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:56.765 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:23:56.765 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:23:56.765 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:23:56.765 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:23:56.766 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:23:56.766 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:23:56.766 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:23:56.766 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:23:56.767 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:23:56.767 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:23:56.767 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:56.767 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:56.767 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:23:56.768 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:23:56.768 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:23:56.768 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:23:56.768 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:23:56.769 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:23:56.769 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:23:56.770 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.336676_339.1 2021-09-17T10:24:01.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:01.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:01.773 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:24:01.824 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:24:01.824 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:24:01.937 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:24:01.938 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:24:01.938 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:24:01.938 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:24:01.938 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:24:01.939 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:24:01.939 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:01.939 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:01.939 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:24:01.940 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:01.940 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:24:01.940 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:01.940 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:01.941 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:24:01.941 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:01.941 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:24:01.941 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:01.941 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:24:01.942 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:01.942 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:01.942 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:01.942 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:01.943 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:24:01.943 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:01.943 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:01.943 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.099168_329.1", 2021-09-17T10:24:01.943 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:01.944 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:01.944 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:01.944 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.099168Z", 2021-09-17T10:24:01.944 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:01.945 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:01.945 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:01.945 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:01.945 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:24:01.946 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:01.946 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:01.946 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:01.946 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:01.947 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:01.947 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:01.947 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:01.947 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:01.948 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:01.948 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:01.948 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:01.948 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:24:01.949 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:01.949 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:01.949 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:01.949 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:01.950 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:01.950 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:01.950 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:01.950 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:01.950 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:01.951 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:01.951 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:24:01.951 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:01.951 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:24:01.952 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:24:01.952 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:24:01.952 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:01.952 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:24:01.953 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.099168_329.1 2021-09-17T10:24:06.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:06.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:06.956 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:24:07.100 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:24:07.101 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:24:07.633 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:24:07.634 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:24:07.634 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:24:07.635 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:24:07.635 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:24:07.635 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:24:07.635 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:07.635 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:07.636 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:24:07.636 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:07.636 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:24:07.636 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:07.637 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:07.637 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:24:07.637 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:07.637 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:24:07.638 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:07.638 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:24:07.638 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:07.638 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:07.639 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:07.639 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:07.639 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:24:07.639 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:07.640 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:07.640 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:24:07.640 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:07.640 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:07.640 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:07.641 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:24:07.641 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:07.641 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:07.641 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:07.642 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:07.642 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:24:07.643 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:07.643 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:07.643 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:07.643 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:07.644 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:07.644 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:07.644 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:07.644 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:07.645 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:07.645 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:07.645 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:07.645 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:24:07.646 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:07.646 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:07.646 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:07.647 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:07.647 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:07.647 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:07.647 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:07.647 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:07.648 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:07.648 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:07.648 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:24:07.648 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:07.649 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:24:07.649 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:24:07.649 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:24:07.649 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:07.650 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:24:07.650 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:24:12.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:12.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:12.653 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:24:12.694 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:24:12.694 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:24:12.810 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:24:12.811 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:24:12.811 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:24:12.811 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:24:12.811 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:24:12.812 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:24:12.812 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:12.812 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:12.812 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:24:12.813 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:12.813 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:24:12.813 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:12.813 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:12.813 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:24:12.814 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:12.814 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:24:12.814 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:12.814 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:24:12.815 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:12.815 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:12.815 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:12.815 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:12.816 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:24:12.816 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:12.816 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:12.816 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:24:12.816 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:12.817 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:12.817 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:12.817 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:24:12.817 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:12.818 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:12.818 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:12.818 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:12.818 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:24:12.818 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:12.819 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:12.819 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:12.819 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:12.819 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:12.820 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:12.820 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:12.820 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:12.820 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:12.820 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:12.821 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:12.821 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:24:12.821 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:12.821 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:12.822 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:12.822 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:12.822 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:12.822 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:12.822 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:12.823 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:12.823 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:12.823 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:12.823 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:24:12.824 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:12.824 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:24:12.824 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:24:12.824 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:24:12.824 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:12.825 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:24:12.826 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.009924_325.1 2021-09-17T10:24:17.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:17.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:17.828 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:24:17.892 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:24:17.893 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:24:18.060 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:24:18.060 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:24:18.061 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:24:18.061 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:24:18.061 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:24:18.062 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:24:18.062 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:18.062 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:18.063 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:24:18.063 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:18.063 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:24:18.063 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:18.063 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:18.064 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:24:18.064 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:18.064 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:24:18.064 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:18.065 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:24:18.065 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:18.065 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:18.065 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:18.065 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:18.066 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:24:18.066 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:18.070 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:18.071 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:24:18.071 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:18.071 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:18.071 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:18.072 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:24:18.072 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:18.072 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:18.072 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:18.073 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:18.073 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:24:18.073 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:18.073 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:18.074 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:18.074 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:18.074 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:18.074 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:18.074 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:18.075 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:18.075 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:18.075 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:18.075 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:18.076 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:24:18.076 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:18.076 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:18.076 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:18.076 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:18.077 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:18.077 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:18.077 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:18.077 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:18.077 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:18.078 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:18.078 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:24:18.078 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:18.078 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:24:18.079 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:24:18.079 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:24:18.079 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:18.079 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:24:18.081 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:24:23.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:23.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:23.083 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:24:23.161 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:24:23.161 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:24:23.526 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:24:23.527 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:24:23.527 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:24:23.527 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:24:23.528 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:24:23.528 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:24:23.528 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:23.528 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:23.529 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:24:23.529 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:23.529 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:24:23.529 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:23.529 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:23.530 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:24:23.530 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:23.530 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:24:23.530 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:23.531 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:24:23.531 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:23.531 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:23.531 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:23.532 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:23.532 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:24:23.532 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:23.532 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:23.533 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:24:23.533 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:23.533 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:23.533 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:23.534 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:24:23.534 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:23.534 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:23.534 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:23.535 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:23.535 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:24:23.535 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:23.535 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:23.535 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:23.536 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:23.536 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:23.536 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:23.536 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:23.537 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:23.537 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:23.537 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:23.537 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:23.538 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:24:23.538 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:23.538 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:23.538 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:23.538 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:23.539 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:23.539 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:23.539 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:23.539 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:23.540 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:23.540 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:23.540 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:24:23.540 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:23.541 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:24:23.541 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:24:23.541 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:24:23.541 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:23.541 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:24:23.543 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.946914_319.1 2021-09-17T10:24:28.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:28.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:28.545 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:24:28.602 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:24:28.603 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:24:28.722 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:24:28.723 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:24:28.723 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:24:28.723 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:24:28.724 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:24:28.724 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:24:28.724 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:28.725 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:28.725 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:24:28.725 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:28.725 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:24:28.725 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:28.726 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:28.726 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:24:28.726 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:28.726 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:24:28.727 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:28.727 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:24:28.727 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:28.728 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:28.728 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:28.728 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:28.729 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:24:28.729 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:28.729 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:28.729 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:24:28.730 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:28.730 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:28.730 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:28.730 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:24:28.731 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:28.731 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:28.731 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:28.731 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:28.731 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:24:28.732 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:28.732 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:28.732 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:28.733 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:28.733 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:28.734 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:28.734 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:28.734 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:28.734 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:28.735 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:28.735 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:28.735 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:24:28.735 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:28.735 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:28.736 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:28.736 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:28.736 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:28.736 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:28.737 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:28.737 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:28.737 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:28.737 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:28.738 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:24:28.738 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:28.738 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:24:28.738 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:24:28.738 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:24:28.739 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:28.739 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:24:28.740 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.946914_319.1 2021-09-17T10:24:33.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:33.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:33.743 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:24:33.823 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:24:33.823 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:24:34.087 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:24:34.088 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:24:34.088 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:24:34.088 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:24:34.089 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:24:34.089 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:24:34.089 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:34.089 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:34.090 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:24:34.090 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:34.090 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:24:34.090 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:34.091 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:34.091 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:24:34.091 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:34.091 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:24:34.091 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:34.092 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:24:34.092 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:34.092 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:34.092 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:34.093 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:34.093 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:24:34.093 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:34.093 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:34.094 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:24:34.094 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:34.094 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:34.094 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:34.094 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:24:34.095 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:34.095 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:34.095 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:34.095 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:34.096 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:24:34.096 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:34.096 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:34.096 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:34.097 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:34.097 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:34.097 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:34.097 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:34.097 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:34.098 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:34.098 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:34.098 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:34.098 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:24:34.099 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:34.099 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:34.099 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:34.099 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:34.100 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:34.100 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:34.100 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:34.100 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:34.100 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:34.101 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:34.101 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:24:34.101 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:34.101 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:24:34.101 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:24:34.102 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:24:34.102 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:34.102 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:24:34.103 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:24:39.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:39.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:39.106 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:24:39.150 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:24:39.150 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:24:39.264 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:24:39.264 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:24:39.265 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:24:39.265 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:24:39.265 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:24:39.265 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:24:39.266 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:39.266 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:39.266 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:24:39.267 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:39.267 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:24:39.267 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:39.267 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:39.268 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:24:39.268 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:39.268 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:24:39.268 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:39.269 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:24:39.269 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:39.269 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:39.269 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:39.270 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:39.270 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:24:39.271 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:39.271 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:39.271 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.249312_334.1", 2021-09-17T10:24:39.272 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:39.272 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:39.272 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:39.272 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.249312Z", 2021-09-17T10:24:39.273 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:39.273 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:39.273 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:39.273 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:39.273 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:24:39.274 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:39.274 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:39.274 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:39.275 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:39.275 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:39.275 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:39.275 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:39.276 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:39.276 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:39.276 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:39.277 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:39.277 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:24:39.277 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:39.277 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:39.278 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:39.279 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:39.279 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:39.279 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:39.279 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:39.280 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:39.280 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:39.280 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:39.280 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:24:39.280 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:39.281 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:24:39.281 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:24:39.281 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:24:39.281 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:39.282 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:24:39.283 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.249312_334.1 2021-09-17T10:24:44.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:44.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:44.288 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:24:44.349 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:24:44.350 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:24:44.506 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:24:44.507 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:24:44.507 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:24:44.507 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:24:44.508 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:24:44.508 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:24:44.508 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:44.508 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:44.509 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:24:44.509 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:44.509 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:24:44.509 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:44.510 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:44.510 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:24:44.510 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:44.510 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:24:44.510 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:44.511 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:24:44.511 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:44.511 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:44.511 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:44.512 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:44.512 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:24:44.512 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:44.512 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:44.513 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:24:44.513 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:44.513 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:44.513 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:44.514 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:24:44.514 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:44.514 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:44.514 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:44.514 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:44.515 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:24:44.515 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:44.515 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:44.515 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:44.516 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:44.516 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:44.516 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:44.516 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:44.516 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:44.517 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:44.517 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:44.517 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:44.518 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:24:44.518 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:44.518 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:44.519 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:44.519 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:44.519 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:44.519 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:44.520 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:44.520 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:44.520 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:44.520 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:44.520 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:24:44.521 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:44.522 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:24:44.522 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:24:44.522 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:24:44.523 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:44.523 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:24:44.525 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.946914_319.1 2021-09-17T10:24:49.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:49.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:49.527 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:24:49.589 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:24:49.589 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:24:49.711 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:24:49.712 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:24:49.712 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:24:49.713 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:24:49.713 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:24:49.714 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:24:49.714 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:49.715 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:49.715 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:24:49.716 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:49.716 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:24:49.716 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:49.717 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:49.717 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:24:49.718 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:49.718 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:24:49.719 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:49.719 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:24:49.719 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:49.720 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:49.720 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:49.721 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:49.721 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:24:49.722 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:49.722 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:49.722 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:24:49.723 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:49.723 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:49.724 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:49.724 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:24:49.725 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:49.725 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:49.725 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:49.726 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:49.726 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:24:49.727 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:49.727 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:49.727 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:49.728 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:49.728 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:49.729 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:49.729 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:49.730 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:49.730 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:49.731 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:49.731 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:49.731 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:24:49.732 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:49.732 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:49.733 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:49.733 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:49.733 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:49.734 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:49.734 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:49.735 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:49.735 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:49.736 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:49.736 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:24:49.736 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:49.737 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:24:49.737 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:24:49.738 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:24:49.738 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:49.739 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:24:49.741 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.946914_319.1 2021-09-17T10:24:54.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:54.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:54.744 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:24:54.796 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:24:54.797 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:24:54.885 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:24:54.885 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:24:54.886 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:24:54.886 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:24:54.886 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:24:54.886 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:24:54.887 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:54.887 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:54.887 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:24:54.888 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:54.888 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:24:54.888 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:54.888 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:54.888 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:24:54.889 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:54.889 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:24:54.889 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:54.889 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:24:54.889 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:54.890 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:54.890 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:54.890 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:54.891 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:24:54.891 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:54.891 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:54.891 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:24:54.891 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:54.892 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:54.892 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:54.892 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:24:54.892 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:54.893 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:54.893 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:54.893 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:54.893 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:24:54.894 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:54.894 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:54.894 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:54.895 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:54.895 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:54.895 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:54.895 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:54.896 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:54.896 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:54.896 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:54.896 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:24:54.897 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:24:54.897 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:24:54.897 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:24:54.897 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:24:54.898 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:24:54.898 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:24:54.898 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:24:54.898 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:24:54.898 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:24:54.899 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:54.899 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:54.899 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:24:54.899 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:24:54.900 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:24:54.900 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:24:54.900 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:24:54.900 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:24:54.901 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:24:54.902 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873652.954369_321.1 2021-09-17T10:24:59.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:59.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:24:59.905 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:24:59.976 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:24:59.976 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:00.370 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:25:00.370 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:25:00.371 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:25:00.371 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:25:00.371 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:25:00.371 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:25:00.371 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:00.372 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:00.372 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:25:00.372 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:00.372 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:25:00.373 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:00.373 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:00.373 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:00.373 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:00.374 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:25:00.374 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:00.374 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:25:00.374 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:00.374 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:00.375 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:00.375 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:00.375 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:25:00.375 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:00.376 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:00.376 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:25:00.376 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:00.376 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:00.376 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:00.377 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:25:00.377 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:00.377 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:00.377 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:00.378 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:00.378 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:25:00.378 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:00.378 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:00.379 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:00.379 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:00.379 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:00.379 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:00.379 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:00.380 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:00.380 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:00.380 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:00.380 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:00.381 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:25:00.381 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:00.381 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:00.381 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:00.382 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:00.382 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:00.382 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:00.382 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:00.382 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:00.383 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:00.383 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:00.383 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:25:00.383 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:00.384 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:25:00.384 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:25:00.384 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:25:00.384 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:00.384 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:25:00.385 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.009924_325.1 2021-09-17T10:25:05.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:05.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:05.388 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:25:05.429 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:25:05.430 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:05.546 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:25:05.547 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:25:05.547 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:25:05.547 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:25:05.548 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:25:05.548 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:25:05.548 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:05.548 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:05.549 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:25:05.549 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:05.549 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:25:05.549 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:05.550 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:05.550 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:05.550 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:05.551 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:25:05.551 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:05.551 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:25:05.551 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:05.552 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:05.552 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:05.552 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:05.552 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:25:05.552 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:05.553 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:05.553 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.007158_324.1", 2021-09-17T10:25:05.553 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:05.553 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:05.554 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:05.554 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.007158Z", 2021-09-17T10:25:05.554 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:05.554 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:05.555 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:05.555 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:05.555 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:25:05.555 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:05.555 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:05.556 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:05.556 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:05.556 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:05.556 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:05.557 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:05.557 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:05.557 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:05.557 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:05.558 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:05.558 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:25:05.558 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:05.558 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:05.559 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:05.559 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:05.559 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:05.559 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:05.560 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:05.560 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:05.560 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:05.560 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:05.561 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:25:05.561 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:05.561 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:25:05.561 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:25:05.562 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:25:05.562 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:05.562 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:25:05.564 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631873986.809317_413.1 target=1_1631873653.007158_324.1 2021-09-17T10:25:10.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2021-09-17T10:25:10.568 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-42 2021-09-17T10:25:10.667 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-43 2021-09-17T10:25:10.786 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-44 2021-09-17T10:25:11.120 INFO:tasks.rgw.c1.client.1.smithi018.stdout:2021-09-17T10:25:11.046+0000 7fe5cbfff700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T10:25:11.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:11.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:11.157 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:25:11.162 INFO:tasks.rgw.c2.client.0.smithi050.stdout:2021-09-17T10:25:11.184+0000 7fd533fff700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T10:25:11.190 INFO:tasks.rgw.c1.client.1.smithi018.stdout:2021-09-17T10:25:11.202+0000 7fe5c67f4700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T10:25:11.220 INFO:tasks.rgw.c1.client.1.smithi018.stdout:2021-09-17T10:25:11.238+0000 7fe5c77f6700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T10:25:11.228 INFO:tasks.rgw.c1.client.1.smithi018.stdout:2021-09-17T10:25:11.246+0000 7fe5e57fa700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T10:25:11.329 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:25:11.329 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:11.705 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:25:11.868 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:25:11.868 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:11.869 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:11.869 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:25:11.869 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:11.869 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:11.870 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:25:11.870 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:25:10.992181Z" 2021-09-17T10:25:11.870 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:11.870 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:11.871 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:11.871 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:25:11.871 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:11.871 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:11.872 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:11.872 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:25:11.872 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:11.872 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:25:11.873 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:25:11.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:11.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:11.874 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:25:11.917 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:25:11.918 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:12.039 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:25:12.039 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:25:12.040 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:25:12.040 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:25:12.040 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:25:12.040 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:25:12.041 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:25:12.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:25:12.041 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:25:12.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:25:12.042 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:25:12.042 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:25:12.042 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:25:12.043 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:12.043 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:25:12.044 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:25:12.044 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:25:12.045 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:25:12.045 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:25:12.045 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:25:12.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:25:12.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:25:12.046 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:25:12.046 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:25:12.046 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:25:12.046 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:25:12.047 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:25:12.047 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:25:12.047 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:25:12.047 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:25:12.048 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:25:12.048 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:25:12.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:25:12.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:25:12.048 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:25:12.049 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:25:12.049 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:25:12.049 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:25:12.049 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:25:12.050 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:25:12.050 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:25:12.050 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:25:12.050 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:25:12.050 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:25:12.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:25:12.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:25:12.051 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:25:12.051 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:25:12.052 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:25:12.052 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:25:12.052 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:25:12.053 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:25:12.053 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:25:12.053 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:25:12.054 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:25:12.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:25:12.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:25:12.055 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:25:12.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:25:12.056 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:25:12.056 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:25:12.056 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:25:12.056 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:25:12.057 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:25:12.058 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:25:12.059 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:25:12.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:12.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:12.060 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:25:12.470 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:25:12.471 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:12.557 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:25:12.558 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:25:12.558 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:25:12.558 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:25:12.558 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:25:12.559 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:25:12.559 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:12.559 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:12.559 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:25:12.560 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:12.560 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:25:12.560 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:12.560 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:12.561 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:12.561 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:12.561 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:25:12.561 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:12.562 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:25:12.562 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:12.562 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:12.562 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:12.562 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:12.563 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:25:12.563 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:12.563 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:12.564 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.419688_344.1", 2021-09-17T10:25:12.564 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:12.564 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:12.564 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:12.565 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.419688Z", 2021-09-17T10:25:12.565 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:12.565 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:12.565 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:12.566 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:12.566 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:25:12.566 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:12.566 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:12.566 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:12.567 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:12.567 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:12.567 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:12.567 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:12.568 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:12.568 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:12.568 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:12.568 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:12.569 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:25:12.569 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:12.569 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:12.570 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:12.570 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:12.570 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:12.570 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:12.571 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:12.571 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:12.571 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:12.571 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:12.572 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:25:12.572 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:12.572 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:25:12.572 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:25:12.573 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:25:12.573 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:12.573 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:25:12.574 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.419688_344.1 2021-09-17T10:25:17.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:17.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:17.578 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:25:17.621 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:25:17.621 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:17.740 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:25:17.740 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:25:17.740 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:25:17.740 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:25:17.741 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:25:17.741 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:25:17.741 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:17.741 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:17.742 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:25:17.742 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:17.742 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:25:17.742 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:17.743 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:17.743 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:17.743 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:17.743 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:25:17.744 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:17.744 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:25:17.744 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:17.744 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:17.745 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:17.745 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:17.745 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:25:17.745 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:17.746 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:17.746 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:25:17.746 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:17.746 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:17.747 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:17.747 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:25:17.747 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:17.747 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:17.748 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:17.748 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:17.748 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:25:17.748 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:17.748 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:17.749 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:17.749 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:17.749 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:17.749 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:17.750 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:17.750 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:17.750 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:17.750 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:17.751 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:17.751 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:25:17.751 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:17.751 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:17.751 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:17.752 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:17.752 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:17.752 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:17.752 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:17.753 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:17.753 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:17.753 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:17.753 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:25:17.753 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:17.754 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:25:17.754 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:25:17.754 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:25:17.754 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:17.755 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:25:17.756 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:25:22.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:22.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:22.758 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:25:22.891 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:25:22.892 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:23.228 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:25:23.229 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:25:23.229 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:25:23.230 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:25:23.230 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:25:23.230 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:25:23.230 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:23.231 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:23.231 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:25:23.232 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:23.232 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:25:23.232 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:23.233 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:23.233 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:23.233 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:23.233 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:25:23.234 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:23.234 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:25:23.234 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:23.234 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:23.235 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:23.235 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:23.235 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:25:23.235 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:23.236 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:23.236 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:25:23.236 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:23.236 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:23.237 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:23.237 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:25:23.238 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:23.238 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:23.239 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:23.239 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:23.239 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:25:23.240 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:23.240 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:23.241 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:23.241 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:23.242 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:23.242 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:23.242 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:23.243 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:23.243 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:23.244 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:23.244 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:23.244 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:25:23.245 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:23.245 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:23.246 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:23.246 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:23.247 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:23.247 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:23.247 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:23.248 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:23.248 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:23.248 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:23.249 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:25:23.249 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:23.250 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:25:23.250 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:25:23.250 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:25:23.251 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:23.251 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:25:23.254 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.946914_319.1 2021-09-17T10:25:28.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:28.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:28.257 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:25:28.297 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:25:28.297 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:28.407 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:25:28.407 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:25:28.408 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:25:28.408 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:25:28.409 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:25:28.409 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:25:28.409 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:28.409 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:28.410 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:25:28.410 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:28.410 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:25:28.410 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:28.411 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:28.411 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:28.411 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:28.411 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:25:28.412 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:28.412 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:25:28.412 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:28.412 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:28.413 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:28.413 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:28.413 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:25:28.413 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:28.414 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:28.414 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:25:28.415 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:28.415 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:28.415 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:28.415 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:25:28.416 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:28.416 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:28.416 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:28.416 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:28.417 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:25:28.417 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:28.417 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:28.417 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:28.418 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:28.418 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:28.418 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:28.418 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:28.419 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:28.419 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:28.419 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:28.419 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:28.419 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:25:28.420 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:28.420 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:28.420 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:28.420 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:28.421 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:28.421 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:28.421 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:28.421 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:28.422 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:28.422 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:28.422 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:25:28.422 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:28.423 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:25:28.423 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:25:28.423 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:25:28.423 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:28.424 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:25:28.425 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:25:33.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:33.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:33.428 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:25:33.543 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:25:33.543 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:33.934 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:25:33.934 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:25:33.934 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:25:33.935 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:25:33.935 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:25:33.935 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:25:33.935 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:33.936 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:33.936 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:25:33.936 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:33.936 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:25:33.937 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:33.937 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:33.937 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:33.937 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:33.937 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:25:33.938 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:33.938 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:25:33.938 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:33.938 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:33.939 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:33.939 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:33.939 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:25:33.939 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:33.940 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:33.940 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:25:33.940 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:33.940 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:33.941 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:33.941 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:25:33.942 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:33.942 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:33.942 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:33.942 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:33.943 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:25:33.943 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:33.943 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:33.943 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:33.943 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:33.944 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:33.944 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:33.944 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:33.944 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:33.945 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:33.945 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:33.945 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:33.945 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:25:33.946 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:33.946 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:33.946 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:33.946 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:33.946 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:33.947 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:33.947 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:33.947 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:33.947 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:33.948 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:33.948 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:25:33.948 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:33.948 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:25:33.948 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:25:33.949 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:25:33.949 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:33.949 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:25:33.950 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.009924_325.1 2021-09-17T10:25:38.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:38.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:38.952 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:25:38.996 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:25:38.996 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:39.079 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:25:39.079 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:25:39.080 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:25:39.080 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:25:39.080 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:25:39.081 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:25:39.081 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:39.081 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:39.081 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:25:39.081 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:39.082 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:25:39.082 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:39.082 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:39.083 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:39.083 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:39.083 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:25:39.083 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:39.084 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:25:39.084 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:39.084 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:39.084 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:39.085 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:39.085 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:25:39.085 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:39.085 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:39.086 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.014731_326.1", 2021-09-17T10:25:39.086 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:39.086 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:39.086 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:39.087 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.014731Z", 2021-09-17T10:25:39.087 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:39.087 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:39.087 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:39.087 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:39.088 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:25:39.088 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:39.088 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:39.088 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:39.089 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:39.089 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:39.089 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:39.089 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:39.090 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:39.090 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:39.090 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:39.090 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:39.091 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:25:39.091 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:39.091 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:39.091 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:39.092 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:39.092 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:39.092 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:39.092 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:39.093 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:39.093 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:39.093 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:39.093 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:25:39.093 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:39.094 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:25:39.094 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:25:39.094 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:25:39.094 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:39.095 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:25:39.096 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.014731_326.1 2021-09-17T10:25:44.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:44.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:44.100 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:25:44.143 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:25:44.143 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:44.252 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:25:44.253 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:25:44.253 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:25:44.253 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:25:44.253 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:25:44.254 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:25:44.254 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:44.255 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:44.255 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:25:44.255 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:44.256 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:25:44.256 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:44.257 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:44.257 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:44.257 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:44.258 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:25:44.258 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:44.258 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:25:44.259 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:44.259 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:44.260 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:44.260 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:44.260 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:25:44.260 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:44.261 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:44.261 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:25:44.261 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:44.261 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:44.262 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:44.262 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:25:44.262 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:44.262 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:44.263 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:44.263 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:44.263 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:25:44.263 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:44.264 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:44.264 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:44.264 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:44.264 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:44.264 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:44.265 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:44.265 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:44.265 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:44.265 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:44.266 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:44.266 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:25:44.266 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:44.267 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:44.267 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:44.267 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:44.268 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:44.268 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:44.268 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:44.268 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:44.268 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:44.269 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:44.269 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:25:44.269 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:44.269 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:25:44.270 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:25:44.270 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:25:44.270 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:44.270 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:25:44.272 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:25:49.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:49.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:49.274 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:25:49.342 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:25:49.342 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:49.648 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:25:49.648 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:25:49.649 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:25:49.649 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:25:49.649 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:25:49.650 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:25:49.650 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:49.650 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:49.650 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:25:49.651 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:49.651 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:25:49.651 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:49.651 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:49.652 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:49.652 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:49.652 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:25:49.652 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:49.653 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:25:49.653 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:49.653 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:49.653 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:49.654 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:49.654 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:25:49.654 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:49.654 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:49.655 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.959198_322.1", 2021-09-17T10:25:49.655 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:49.655 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:49.655 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:49.656 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.959198Z", 2021-09-17T10:25:49.656 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:49.656 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:49.656 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:49.657 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:49.657 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:25:49.657 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:49.657 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:49.658 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:49.658 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:49.658 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:49.659 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:49.659 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:49.660 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:49.660 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:49.660 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:49.660 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:49.667 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:25:49.667 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:49.668 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:49.668 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:49.669 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:49.669 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:49.670 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:49.670 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:49.671 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:49.671 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:49.671 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:49.671 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:25:49.672 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:49.673 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:25:49.673 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:25:49.674 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:25:49.674 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:49.674 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:25:49.678 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.959198_322.1 2021-09-17T10:25:54.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:54.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:54.681 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:25:54.746 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:25:54.746 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:25:54.866 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:25:54.867 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:25:54.867 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:25:54.867 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:25:54.867 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:25:54.868 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:25:54.868 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:54.868 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:54.868 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:25:54.869 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:54.869 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:25:54.869 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:54.869 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:54.870 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:25:54.870 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:54.870 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:25:54.870 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:54.871 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:25:54.871 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:54.871 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:54.871 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:54.872 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:54.872 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:25:54.872 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:54.872 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:54.873 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:25:54.873 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:54.873 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:54.874 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:54.874 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:25:54.874 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:54.874 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:54.875 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:54.875 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:54.875 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:25:54.875 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:54.876 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:54.876 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:54.876 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:54.876 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:54.877 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:54.877 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:54.877 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:54.877 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:54.877 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:54.878 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:25:54.878 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:25:54.878 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:25:54.878 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:25:54.879 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:25:54.879 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:25:54.879 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:25:54.880 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:25:54.880 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:25:54.880 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:25:54.880 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:54.880 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:54.881 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:25:54.881 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:25:54.881 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:25:54.881 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:25:54.882 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:25:54.882 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:25:54.882 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:25:54.883 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.009924_325.1 2021-09-17T10:25:59.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:59.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:25:59.886 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:26:00.003 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:26:00.004 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:26:00.467 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:26:00.467 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:26:00.467 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:26:00.468 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:26:00.468 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:26:00.468 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:26:00.468 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:00.469 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:00.469 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:26:00.469 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:00.469 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:26:00.470 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:00.470 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:00.470 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:26:00.470 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:00.470 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:26:00.471 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:00.471 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:26:00.471 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:00.471 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:00.472 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:00.472 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:00.472 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:26:00.472 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:00.472 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:00.473 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:26:00.473 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:00.473 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:00.473 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:00.474 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:26:00.474 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:00.474 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:00.474 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:00.474 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:00.475 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:26:00.475 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:00.475 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:00.475 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:00.476 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:00.476 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:00.476 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:00.476 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:00.476 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:00.477 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:00.477 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:00.477 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:00.477 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:26:00.478 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:00.478 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:00.478 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:00.478 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:00.478 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:00.479 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:00.479 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:00.479 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:00.479 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:00.480 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:00.480 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:26:00.480 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:00.480 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:26:00.480 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:26:00.481 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:26:00.481 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:00.481 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:26:00.483 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:26:05.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:05.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:05.486 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:26:05.545 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:26:05.546 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:26:05.632 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:26:05.633 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:26:05.633 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:26:05.633 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:26:05.633 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:26:05.634 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:26:05.634 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:05.634 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:05.634 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:26:05.635 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:05.635 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:26:05.635 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:05.635 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:05.635 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:26:05.636 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:05.636 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:26:05.636 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:05.636 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:26:05.637 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:05.637 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:05.637 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:05.637 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:05.638 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:26:05.638 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:05.638 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:05.638 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:26:05.638 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:05.639 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:05.639 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:05.639 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:26:05.639 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:05.640 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:05.640 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:05.640 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:05.640 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:26:05.640 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:05.641 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:05.641 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:05.641 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:05.641 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:05.642 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:05.642 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:05.642 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:05.642 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:05.643 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:05.643 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:05.643 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:26:05.643 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:05.643 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:05.644 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:05.644 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:05.644 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:05.644 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:05.645 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:05.645 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:05.645 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:05.645 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:05.645 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:26:05.646 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:05.646 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:26:05.646 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:26:05.646 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:26:05.647 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:05.647 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:26:05.648 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.946914_319.1 2021-09-17T10:26:10.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:10.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:10.651 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:26:10.695 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:26:10.695 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:26:10.818 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:26:10.818 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:26:10.818 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:26:10.819 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:26:10.819 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:26:10.819 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:26:10.819 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:10.820 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:10.820 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:26:10.820 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:10.820 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:26:10.820 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:10.821 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:10.821 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:26:10.821 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:10.821 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:26:10.821 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:10.822 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:26:10.822 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:10.822 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:10.822 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:10.823 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:10.823 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:26:10.823 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:10.823 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:10.824 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:26:10.824 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:10.824 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:10.824 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:10.825 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:26:10.825 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:10.825 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:10.825 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:10.826 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:10.826 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:26:10.826 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:10.826 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:10.827 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:10.827 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:10.827 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:10.827 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:10.827 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:10.828 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:10.828 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:10.828 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:10.828 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:10.829 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:26:10.829 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:10.829 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:10.829 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:10.829 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:10.830 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:10.830 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:10.830 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:10.830 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:10.831 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:10.831 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:10.831 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:26:10.831 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:10.832 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:26:10.832 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:26:10.832 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:26:10.832 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:10.832 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:26:10.834 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.952206_320.1 2021-09-17T10:26:15.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:15.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:15.837 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:26:15.919 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:26:15.920 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:26:16.323 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:26:16.323 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:26:16.323 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:26:16.324 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:26:16.324 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:26:16.324 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:26:16.324 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:16.324 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:16.325 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:26:16.325 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:16.325 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:26:16.325 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:16.326 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:16.326 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:26:16.326 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:16.326 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:26:16.327 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:16.327 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:26:16.327 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:16.327 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:16.328 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:16.328 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:16.328 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:26:16.328 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:16.329 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:16.329 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.336676_339.1", 2021-09-17T10:26:16.329 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:16.329 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:16.329 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:16.330 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.336676Z", 2021-09-17T10:26:16.330 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:16.330 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:16.330 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:16.331 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:16.331 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:26:16.331 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:16.331 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:16.331 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:16.332 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:16.332 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:16.332 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:16.332 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:16.333 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:16.333 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:16.333 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:16.333 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:16.333 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:26:16.334 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:16.334 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:16.334 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:16.335 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:16.335 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:16.335 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:16.335 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:16.336 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:16.336 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:16.336 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:16.336 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:26:16.337 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:16.337 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:26:16.337 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:26:16.337 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:26:16.337 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:16.338 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:26:16.339 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.336676_339.1 2021-09-17T10:26:21.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:21.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:21.342 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:26:21.382 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:26:21.382 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:26:21.496 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:26:21.497 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:26:21.497 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:26:21.497 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:26:21.497 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:26:21.498 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:26:21.498 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:21.498 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:21.498 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:26:21.499 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:21.499 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:26:21.499 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:21.499 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:21.500 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:26:21.500 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:21.500 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:26:21.500 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:21.500 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:26:21.501 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:21.501 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:21.501 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:21.501 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:21.501 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:26:21.502 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:21.502 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:21.502 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:26:21.502 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:21.503 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:21.503 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:21.503 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:26:21.503 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:21.503 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:21.504 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:21.504 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:21.504 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:26:21.504 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:21.505 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:21.505 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:21.505 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:21.505 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:21.505 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:21.506 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:21.506 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:21.506 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:21.506 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:21.506 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:21.507 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:26:21.507 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:21.507 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:21.507 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:21.508 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:21.508 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:21.508 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:21.508 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:21.508 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:21.509 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:21.509 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:21.509 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:26:21.510 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:21.510 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:26:21.510 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:26:21.510 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:26:21.510 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:21.511 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:26:21.512 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.952206_320.1 2021-09-17T10:26:26.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:26.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:26.515 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:26:26.559 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:26:26.559 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:26:26.687 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:26:26.688 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:26:26.688 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:26:26.688 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:26:26.689 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:26:26.689 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:26:26.689 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:26.690 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:26.690 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:26:26.690 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:26.690 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:26:26.690 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:26.691 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:26.691 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:26:26.691 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:26.691 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:26:26.692 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:26.692 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:26:26.692 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:26.692 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:26.692 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:26.693 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:26.693 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:26:26.693 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:26.693 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:26.694 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:26:26.694 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:26.694 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:26.694 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:26.694 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:26:26.695 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:26.695 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:26.695 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:26.695 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:26.696 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:26:26.696 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:26.696 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:26.696 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:26.697 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:26.697 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:26.697 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:26.697 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:26.697 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:26.698 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:26.698 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:26.698 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:26.698 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:26:26.699 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:26.699 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:26.699 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:26.699 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:26.700 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:26.700 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:26.700 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:26.700 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:26.701 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:26.701 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:26.701 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:26:26.701 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:26.702 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:26:26.702 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:26:26.702 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:26:26.702 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:26.702 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:26:26.704 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.952206_320.1 2021-09-17T10:26:31.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:31.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:31.707 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:26:31.804 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:26:31.804 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:26:32.117 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:26:32.117 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:26:32.118 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:26:32.118 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:26:32.118 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:26:32.118 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:26:32.119 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:32.119 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:32.119 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:26:32.119 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:32.120 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:26:32.120 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:32.120 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:32.120 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:26:32.120 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:32.121 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:26:32.121 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:32.121 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:26:32.121 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:32.122 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:32.122 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:32.122 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:32.122 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:26:32.123 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:32.123 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:32.123 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:26:32.123 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:32.124 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:32.124 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:32.124 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:26:32.124 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:32.124 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:32.125 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:32.125 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:32.125 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:26:32.125 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:32.126 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:32.126 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:32.126 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:32.126 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:32.127 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:32.127 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:32.127 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:32.127 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:32.127 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:32.128 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:32.128 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:26:32.128 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:32.128 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:32.129 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:32.129 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:32.129 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:32.129 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:32.129 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:32.130 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:32.130 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:32.130 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:32.130 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:26:32.131 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:32.131 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:26:32.131 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:26:32.131 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:26:32.131 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:32.132 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:26:32.132 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:26:37.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:37.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:37.135 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:26:37.178 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:26:37.178 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:26:37.294 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:26:37.294 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:26:37.295 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:26:37.295 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:26:37.295 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:26:37.295 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:26:37.296 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:37.296 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:37.296 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:26:37.296 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:37.297 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:26:37.297 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:37.297 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:37.297 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:26:37.297 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:37.298 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:26:37.298 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:37.298 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:26:37.298 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:37.299 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:37.299 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:37.299 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:37.299 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:26:37.299 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:37.300 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:37.300 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.240117_333.1", 2021-09-17T10:26:37.300 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:37.301 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:37.301 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:37.301 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.240117Z", 2021-09-17T10:26:37.301 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:37.301 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:37.302 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:37.302 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:37.302 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:26:37.302 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:37.303 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:37.303 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:37.303 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:37.304 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:37.304 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:37.304 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:37.304 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:37.304 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:37.305 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:37.305 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:37.305 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:26:37.305 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:37.306 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:37.306 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:37.306 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:37.306 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:37.306 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:37.307 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:37.307 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:37.307 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:37.307 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:37.308 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:26:37.308 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:37.308 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:26:37.308 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:26:37.308 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:26:37.309 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:37.309 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:26:37.310 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.240117_333.1 2021-09-17T10:26:42.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:42.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:42.313 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:26:42.387 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:26:42.387 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:26:42.779 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:26:42.779 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:26:42.780 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:26:42.780 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:26:42.780 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:26:42.780 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:26:42.781 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:42.781 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:42.781 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:26:42.781 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:42.782 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:26:42.782 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:42.782 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:42.782 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:26:42.783 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:42.783 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:26:42.783 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:42.783 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:26:42.784 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:42.784 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:42.784 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:42.784 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:42.785 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:26:42.785 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:42.785 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:42.785 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:26:42.785 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:42.786 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:42.786 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:42.786 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:26:42.786 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:42.787 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:42.787 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:42.787 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:42.788 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:26:42.788 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:42.788 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:42.788 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:42.789 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:42.789 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:42.789 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:42.790 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:42.790 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:42.790 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:42.790 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:42.791 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:42.791 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:26:42.791 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:42.791 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:42.792 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:42.792 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:42.792 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:42.792 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:42.793 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:42.793 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:42.793 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:42.793 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:42.793 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:26:42.794 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:42.794 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:26:42.794 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:26:42.794 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:26:42.795 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:42.795 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:26:42.795 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.952206_320.1 2021-09-17T10:26:47.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:47.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:47.798 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:26:47.840 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:26:47.840 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:26:47.960 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:26:47.961 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:26:47.961 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:26:47.961 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:26:47.961 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:26:47.962 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:26:47.962 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:47.962 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:47.962 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:26:47.963 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:47.963 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:26:47.963 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:47.963 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:47.964 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:26:47.964 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:47.964 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:26:47.964 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:47.965 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:26:47.965 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:47.965 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:47.966 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:47.966 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:47.966 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:26:47.966 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:47.967 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:47.967 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.256150_335.1", 2021-09-17T10:26:47.967 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:47.967 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:47.968 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:47.968 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.256150Z", 2021-09-17T10:26:47.968 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:47.968 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:47.969 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:47.969 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:47.969 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:26:47.969 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:47.969 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:47.970 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:47.970 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:47.970 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:47.970 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:47.971 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:47.971 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:47.971 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:47.971 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:47.971 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:47.972 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:26:47.972 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:47.972 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:47.972 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:47.973 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:47.973 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:47.973 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:47.973 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:47.974 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:47.974 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:47.974 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:47.974 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:26:47.974 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:47.975 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:26:47.975 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:26:47.975 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:26:47.975 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:47.976 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:26:47.977 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.256150_335.1 2021-09-17T10:26:52.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:52.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:52.980 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:26:53.023 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:26:53.023 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:26:53.214 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:26:53.215 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:26:53.215 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:26:53.215 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:26:53.216 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:26:53.216 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:26:53.216 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:53.216 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:53.216 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:26:53.217 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:53.217 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:26:53.217 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:53.217 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:53.218 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:26:53.218 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:53.218 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:26:53.218 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:53.219 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:26:53.219 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:53.219 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:53.219 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:53.219 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:53.220 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:26:53.220 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:53.220 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:53.220 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:26:53.221 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:53.221 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:53.221 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:53.221 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:26:53.222 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:53.222 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:53.222 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:53.222 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:53.222 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:26:53.223 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:53.223 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:53.223 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:53.223 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:53.224 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:53.224 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:53.224 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:53.224 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:53.224 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:53.225 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:53.225 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:53.225 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:26:53.225 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:53.226 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:53.226 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:53.226 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:53.226 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:53.227 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:53.227 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:53.227 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:53.227 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:53.227 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:53.228 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:26:53.228 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:53.228 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:26:53.228 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:26:53.229 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:26:53.229 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:53.229 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:26:53.230 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.952206_320.1 2021-09-17T10:26:58.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:58.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:26:58.233 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:26:58.321 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:26:58.322 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:26:58.529 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:26:58.529 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:26:58.529 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:26:58.530 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:26:58.530 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:26:58.530 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:26:58.530 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:58.531 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:58.531 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:26:58.531 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:58.531 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:26:58.532 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:58.532 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:58.532 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:26:58.532 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:58.533 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:26:58.533 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:58.533 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:26:58.533 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:58.533 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:58.534 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:58.534 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:58.534 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:26:58.534 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:58.535 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:58.535 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:26:58.535 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:58.535 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:58.536 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:58.536 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:26:58.536 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:58.536 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:58.537 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:58.537 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:58.537 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:26:58.537 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:58.537 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:58.538 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:58.538 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:58.538 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:58.538 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:58.539 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:58.539 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:58.539 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:58.539 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:58.539 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:26:58.540 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:26:58.540 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:26:58.540 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:26:58.540 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:26:58.541 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:26:58.541 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:26:58.541 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:26:58.541 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:26:58.541 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:26:58.542 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:58.542 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:58.542 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:26:58.542 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:26:58.543 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:26:58.543 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:26:58.543 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:26:58.543 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:26:58.543 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:26:58.545 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:27:03.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:03.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:03.547 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:27:03.591 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:27:03.591 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:27:03.713 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:27:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:27:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:27:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:27:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:27:03.715 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:27:03.715 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:03.715 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:03.715 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:27:03.715 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:03.716 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:27:03.716 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:03.716 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:03.716 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:27:03.717 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:03.717 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:27:03.717 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:03.717 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:27:03.718 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:03.718 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:03.718 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:03.718 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:03.719 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:27:03.719 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:03.719 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:03.719 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:27:03.719 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:03.720 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:03.720 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:03.720 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:27:03.720 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:03.721 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:03.721 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:03.721 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:03.721 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:27:03.722 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:03.722 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:03.722 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:03.722 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:03.722 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:03.723 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:03.723 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:03.723 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:03.723 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:03.724 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:03.724 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:03.724 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:27:03.724 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:03.724 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:03.725 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:03.725 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:03.725 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:03.725 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:03.726 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:03.726 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:03.726 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:03.726 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:03.726 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:27:03.727 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:03.727 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:27:03.727 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:27:03.727 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:27:03.728 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:03.728 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:27:03.729 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.952206_320.1 2021-09-17T10:27:08.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:08.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:08.731 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:27:08.803 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:27:08.803 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:27:09.165 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:27:09.166 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:27:09.166 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:27:09.166 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:27:09.167 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:27:09.167 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:27:09.167 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:09.167 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:09.168 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:27:09.168 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:09.168 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:27:09.168 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:09.168 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:09.169 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:27:09.169 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:09.169 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:27:09.169 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:09.170 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:27:09.170 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:09.170 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:09.170 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:09.170 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:09.171 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:27:09.171 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:09.171 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:09.171 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:27:09.172 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:09.172 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:09.172 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:09.172 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:27:09.173 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:09.173 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:09.173 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:09.173 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:09.173 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:27:09.174 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:09.174 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:09.174 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:09.174 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:09.175 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:09.175 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:09.175 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:09.175 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:09.175 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:09.176 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:09.176 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:09.176 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:27:09.176 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:09.177 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:09.177 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:09.177 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:09.177 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:09.177 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:09.178 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:09.178 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:09.178 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:09.178 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:09.179 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:27:09.179 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:09.179 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:27:09.179 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:27:09.179 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:27:09.180 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:09.180 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:27:09.181 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.922243_318.1 2021-09-17T10:27:14.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:14.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:14.184 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:27:14.228 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:27:14.228 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:27:14.338 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:27:14.338 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:27:14.338 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:27:14.339 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:27:14.339 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:27:14.339 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:27:14.339 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:14.340 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:14.340 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:27:14.340 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:14.340 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:27:14.340 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:14.341 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:14.341 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:27:14.341 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:14.341 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:27:14.342 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:14.342 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:27:14.342 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:14.342 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:14.343 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:14.343 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:14.343 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:27:14.343 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:14.344 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:14.344 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:27:14.344 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:14.344 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:14.344 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:14.345 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:27:14.345 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:14.345 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:14.345 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:14.346 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:14.347 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:27:14.347 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:14.347 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:14.348 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:14.348 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:14.348 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:14.348 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:14.348 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:14.349 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:14.349 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:14.349 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:14.349 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:14.350 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:27:14.350 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:14.350 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:14.350 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:14.350 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:14.351 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:14.351 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:14.351 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:14.351 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:14.352 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:14.352 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:14.352 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:27:14.352 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:14.352 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:27:14.353 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:27:14.353 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:27:14.353 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:14.353 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:27:14.355 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.922243_318.1 2021-09-17T10:27:19.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:19.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:19.357 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:27:19.400 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:27:19.400 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:27:19.573 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:27:19.573 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:27:19.573 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:27:19.574 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:27:19.574 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:27:19.574 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:27:19.574 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:19.575 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:19.575 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:27:19.575 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:19.575 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:27:19.576 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:19.576 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:19.576 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:27:19.576 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:19.576 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:27:19.577 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:19.577 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:27:19.577 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:19.577 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:19.578 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:19.578 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:19.578 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:27:19.578 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:19.578 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:19.579 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:27:19.579 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:19.579 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:19.579 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:19.580 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:27:19.580 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:19.580 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:19.580 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:19.581 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:19.581 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:27:19.581 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:19.581 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:19.582 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:19.582 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:19.582 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:19.582 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:19.583 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:19.583 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:19.583 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:19.583 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:19.584 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:19.584 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:27:19.584 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:19.584 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:19.584 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:19.585 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:19.585 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:19.585 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:19.585 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:19.586 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:19.586 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:19.586 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:19.586 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:27:19.587 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:19.587 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:27:19.587 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:27:19.587 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:27:19.588 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:19.588 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:27:19.589 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.922243_318.1 2021-09-17T10:27:24.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:24.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:24.592 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:27:24.653 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:27:24.654 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:27:24.899 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:27:24.899 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:27:24.899 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:27:24.900 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:27:24.900 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:27:24.900 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:27:24.900 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:24.900 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:24.901 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:27:24.901 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:24.901 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:27:24.901 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:24.902 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:24.902 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:27:24.902 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:24.902 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:27:24.903 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:24.903 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:27:24.903 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:24.903 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:24.904 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:24.904 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:24.904 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:27:24.904 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:24.904 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:24.905 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:27:24.905 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:24.905 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:24.905 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:24.906 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:27:24.906 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:24.906 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:24.906 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:24.907 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:24.907 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:27:24.907 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:24.907 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:24.907 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:24.908 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:24.908 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:24.908 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:24.909 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:24.909 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:24.909 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:24.909 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:24.910 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:24.910 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:27:24.910 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:24.910 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:24.911 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:24.911 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:24.911 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:24.911 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:24.911 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:24.912 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:24.912 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:24.912 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:24.912 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:27:24.913 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:24.913 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:27:24.913 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:27:24.913 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:27:24.913 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:24.914 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:27:24.915 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:27:29.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:29.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:29.918 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:27:29.960 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:27:29.960 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:27:30.078 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:27:30.079 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:27:30.079 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:27:30.079 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:27:30.079 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:27:30.079 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:27:30.080 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:30.080 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:30.080 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:27:30.080 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:30.081 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:27:30.081 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:30.081 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:30.081 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:27:30.081 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:30.082 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:27:30.082 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:30.082 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:27:30.082 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:30.083 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:30.083 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:30.083 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:30.083 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:27:30.084 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:30.084 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:30.084 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:27:30.084 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:30.084 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:30.085 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:30.085 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:27:30.085 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:30.085 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:30.086 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:30.086 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:30.086 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:27:30.086 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:30.086 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:30.087 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:30.087 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:30.087 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:30.087 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:30.088 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:30.088 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:30.088 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:30.088 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:30.088 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:30.089 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:27:30.089 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:30.089 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:30.089 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:30.090 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:30.090 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:30.090 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:30.090 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:30.090 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:30.091 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:30.091 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:30.091 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:27:30.091 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:30.092 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:27:30.092 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:27:30.092 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:27:30.092 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:30.092 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:27:30.094 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.009924_325.1 2021-09-17T10:27:35.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:35.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:35.097 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:27:35.199 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:27:35.199 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:27:35.607 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:27:35.608 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:27:35.608 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:27:35.608 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:27:35.608 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:27:35.609 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:27:35.609 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:35.609 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:35.609 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:27:35.609 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:35.610 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:27:35.610 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:35.610 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:35.610 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:27:35.611 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:35.611 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:27:35.611 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:35.611 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:27:35.612 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:35.612 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:35.612 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:35.612 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:35.613 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:27:35.613 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:35.613 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:35.613 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:27:35.613 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:35.614 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:35.614 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:35.614 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:27:35.614 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:35.615 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:35.615 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:35.615 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:35.615 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:27:35.616 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:35.616 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:35.616 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:35.616 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:35.617 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:35.617 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:35.617 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:35.617 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:35.618 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:35.618 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:35.618 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:35.618 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:27:35.619 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:35.619 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:35.619 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:35.619 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:35.620 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:35.620 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:35.620 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:35.620 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:35.620 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:35.621 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:35.621 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:27:35.621 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:35.621 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:27:35.622 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:27:35.622 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:27:35.622 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:35.622 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:27:35.623 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.009924_325.1 2021-09-17T10:27:40.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:40.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:40.626 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:27:40.670 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:27:40.671 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:27:40.791 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:27:40.791 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:27:40.791 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:27:40.792 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:27:40.792 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:27:40.792 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:27:40.792 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:40.793 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:40.793 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:27:40.793 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:40.793 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:27:40.793 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:40.794 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:40.794 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:27:40.794 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:40.794 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:27:40.795 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:40.795 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:27:40.795 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:40.795 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:40.795 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:40.796 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:40.796 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:27:40.796 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:40.796 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:40.797 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:27:40.797 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:40.797 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:40.797 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:40.797 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:27:40.798 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:40.798 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:40.798 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:40.798 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:40.799 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:27:40.799 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:40.799 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:40.799 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:40.799 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:40.800 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:40.800 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:40.800 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:40.800 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:40.800 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:40.801 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:40.801 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:40.801 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:27:40.801 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:40.802 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:40.802 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:40.802 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:40.802 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:40.803 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:40.803 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:40.803 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:40.803 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:40.803 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:40.804 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:27:40.804 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:40.804 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:27:40.804 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:27:40.804 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:27:40.805 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:40.805 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:27:40.806 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.946914_319.1 2021-09-17T10:27:45.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:45.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:45.809 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:27:45.852 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:27:45.852 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:27:45.972 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:27:45.973 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:27:45.973 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:27:45.973 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:27:45.974 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:27:45.974 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:27:45.974 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:45.974 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:27:45.976 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:45.976 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:45.976 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:27:45.976 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:45.977 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:27:45.977 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:45.977 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:27:45.977 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:45.977 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:45.978 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:45.978 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:45.978 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:27:45.979 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:45.979 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:45.979 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:27:45.979 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:45.980 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:45.980 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:45.980 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:27:45.980 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:45.981 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:45.981 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:45.981 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:45.981 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:27:45.981 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:45.982 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:45.982 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:45.982 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:45.982 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:45.983 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:45.983 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:45.983 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:45.983 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:45.983 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:45.984 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:45.984 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:27:45.984 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:45.984 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:45.985 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:45.985 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:45.985 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:45.985 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:45.986 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:45.986 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:45.986 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:45.986 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:45.986 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:27:45.987 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:45.987 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:27:45.987 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:27:45.987 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:27:45.988 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:45.988 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:27:45.989 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.952206_320.1 2021-09-17T10:27:50.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:50.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:50.992 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:27:51.086 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:27:51.086 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:27:51.425 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:27:51.425 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:27:51.426 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:27:51.426 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:27:51.426 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:27:51.426 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:27:51.427 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:51.427 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:51.427 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:27:51.427 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:51.427 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:27:51.428 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:51.428 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:51.428 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:27:51.428 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:51.429 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:27:51.429 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:51.429 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:27:51.429 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:51.429 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:51.430 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:51.430 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:51.430 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:27:51.430 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:51.431 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:51.431 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.038405_327.1", 2021-09-17T10:27:51.431 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:51.431 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:51.431 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:51.432 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.038405Z", 2021-09-17T10:27:51.432 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:51.432 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:51.432 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:51.432 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:51.433 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:27:51.433 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:51.433 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:51.433 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:51.434 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:51.434 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:51.434 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:51.434 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:51.434 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:51.435 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:51.435 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:51.435 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:51.435 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:27:51.436 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:51.436 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:51.436 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:51.436 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:51.436 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:51.437 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:51.437 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:51.437 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:51.437 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:51.437 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:51.438 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:27:51.438 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:51.438 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:27:51.438 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:27:51.439 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:27:51.439 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:51.439 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:27:51.440 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.038405_327.1 2021-09-17T10:27:56.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:56.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:27:56.443 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:27:56.486 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:27:56.486 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:27:56.608 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:27:56.609 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:27:56.609 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:27:56.609 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:27:56.609 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:27:56.609 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:27:56.610 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:56.610 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:56.610 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:27:56.610 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:56.611 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:27:56.611 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:56.611 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:56.611 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:27:56.612 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:56.612 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:27:56.612 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:56.612 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:27:56.612 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:56.613 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:56.613 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:56.613 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:56.613 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:27:56.614 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:56.614 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:56.614 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.001301_323.1", 2021-09-17T10:27:56.614 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:56.615 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:56.615 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:56.615 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.001301Z", 2021-09-17T10:27:56.615 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:56.615 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:56.616 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:56.616 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:56.616 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:27:56.616 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:56.617 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:56.617 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:56.617 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:56.617 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:56.617 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:56.618 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:56.618 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:56.618 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:56.618 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:56.619 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:27:56.619 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:27:56.619 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:27:56.619 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:27:56.620 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:27:56.620 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:27:56.620 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:27:56.620 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:27:56.620 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:27:56.621 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:27:56.621 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:56.621 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:56.621 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:27:56.622 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:27:56.622 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:27:56.622 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:27:56.622 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:27:56.622 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:27:56.623 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:27:56.624 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.001301_323.1 2021-09-17T10:28:01.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:01.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:01.628 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:28:01.707 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:28:01.707 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:28:02.169 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:28:02.170 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:28:02.170 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:28:02.170 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:28:02.171 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:28:02.171 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:28:02.172 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:02.172 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:02.173 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:28:02.173 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:02.175 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:28:02.175 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:02.175 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:02.175 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:28:02.176 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:02.176 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:28:02.176 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:02.176 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:28:02.177 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:02.177 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:02.177 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:02.177 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:02.178 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:28:02.178 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:02.178 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:02.178 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.014731_326.1", 2021-09-17T10:28:02.179 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:02.179 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:02.179 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:02.179 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.014731Z", 2021-09-17T10:28:02.180 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:02.180 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:02.180 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:02.180 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:02.181 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:28:02.181 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:02.181 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:02.181 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:02.181 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:02.182 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:02.182 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:02.182 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:02.182 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:02.183 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:02.183 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:02.183 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:02.183 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:28:02.184 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:02.184 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:02.184 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:02.184 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:02.184 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:02.185 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:02.185 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:02.185 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:02.185 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:02.185 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:02.186 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:28:02.186 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:02.186 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:28:02.186 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:28:02.187 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:28:02.187 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:02.187 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:28:02.189 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.014731_326.1 2021-09-17T10:28:07.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:07.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:07.192 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:28:07.233 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:28:07.234 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:28:07.326 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:28:07.327 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:28:07.327 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:28:07.327 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:28:07.327 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:28:07.328 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:28:07.328 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:07.328 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:07.328 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:28:07.329 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:07.329 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:28:07.329 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:07.329 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:07.330 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:28:07.330 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:07.330 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:28:07.330 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:07.330 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:28:07.331 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:07.331 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:07.331 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:07.331 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:07.332 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:28:07.332 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:07.332 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:07.332 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.001301_323.1", 2021-09-17T10:28:07.333 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:07.333 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:07.333 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:07.333 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.001301Z", 2021-09-17T10:28:07.333 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:07.334 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:07.334 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:07.334 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:07.334 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:28:07.335 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:07.335 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:07.335 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:07.335 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:07.336 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:07.336 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:07.336 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:07.336 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:07.337 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:07.337 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:07.337 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:07.337 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:28:07.338 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:07.338 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:07.338 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:07.338 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:07.339 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:07.339 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:07.339 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:07.339 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:07.340 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:07.340 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:07.340 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:28:07.340 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:07.341 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:28:07.341 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:28:07.341 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:28:07.341 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:07.342 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:28:07.343 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.001301_323.1 2021-09-17T10:28:12.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:12.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:12.345 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:28:12.423 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:28:12.423 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:28:12.545 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:28:12.546 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:28:12.546 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:28:12.546 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:28:12.547 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:28:12.547 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:28:12.547 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:12.547 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:12.548 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:28:12.548 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:12.548 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:28:12.548 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:12.549 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:12.549 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:28:12.549 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:12.549 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:28:12.549 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:12.550 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:28:12.550 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:12.550 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:12.550 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:12.551 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:12.551 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:28:12.551 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:12.551 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:12.551 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:28:12.552 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:12.552 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:12.552 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:12.552 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:28:12.553 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:12.553 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:12.553 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:12.553 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:12.553 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:28:12.554 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:12.554 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:12.554 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:12.554 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:12.555 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:12.555 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:12.555 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:12.555 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:12.555 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:12.556 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:12.556 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:12.556 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:28:12.556 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:12.557 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:12.557 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:12.557 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:12.557 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:12.557 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:12.558 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:12.558 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:12.558 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:12.558 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:12.559 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:28:12.559 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:12.559 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:28:12.559 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:28:12.559 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:28:12.560 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:12.560 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:28:12.561 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:28:17.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:17.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:17.564 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:28:17.953 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:28:17.954 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:28:18.115 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:28:18.116 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:28:18.116 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:28:18.116 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:28:18.116 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:28:18.116 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:28:18.117 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:18.117 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:18.117 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:28:18.117 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:18.117 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:28:18.118 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:18.118 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:18.118 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:28:18.118 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:18.118 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:28:18.119 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:18.119 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:28:18.119 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:18.119 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:18.120 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:18.120 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:18.120 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:28:18.120 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:18.120 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:18.121 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:28:18.121 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:18.121 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:18.121 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:18.121 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:28:18.122 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:18.122 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:18.122 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:18.122 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:18.122 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:28:18.123 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:18.123 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:18.123 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:18.123 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:18.123 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:18.124 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:18.124 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:18.124 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:18.124 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:18.124 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:18.125 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:18.125 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:28:18.126 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:18.126 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:18.126 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:18.126 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:18.127 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:18.127 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:18.127 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:18.127 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:18.127 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:18.128 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:18.128 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:28:18.128 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:18.128 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:28:18.129 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:28:18.129 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:28:18.129 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:18.129 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:28:18.130 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.952206_320.1 2021-09-17T10:28:23.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:23.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:23.134 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:28:23.173 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:28:23.173 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:28:23.287 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:28:23.287 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:28:23.287 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:28:23.288 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:28:23.288 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:28:23.288 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:28:23.288 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:23.289 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:23.289 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:28:23.289 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:23.289 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:28:23.289 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:23.290 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:23.290 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:28:23.290 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:23.290 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:28:23.291 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:23.291 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:28:23.291 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:23.291 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:23.292 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:23.292 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:23.292 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:28:23.292 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:23.292 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:23.293 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:28:23.293 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:23.293 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:23.293 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:23.293 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:28:23.294 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:23.294 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:23.294 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:23.294 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:23.295 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:28:23.295 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:23.295 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:23.295 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:23.296 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:23.296 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:23.296 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:23.296 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:23.296 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:23.297 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:23.297 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:23.297 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:23.297 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:28:23.297 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:23.298 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:23.298 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:23.298 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:23.298 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:23.298 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:23.299 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:23.299 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:23.299 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:23.299 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:23.300 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:28:23.300 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:23.300 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:28:23.300 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:28:23.300 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:28:23.301 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:23.301 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:28:23.302 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.952206_320.1 2021-09-17T10:28:28.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:28.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:28.304 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:28:28.359 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:28:28.359 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:28:28.728 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:28:28.729 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:28:28.729 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:28:28.729 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:28:28.729 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:28:28.729 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:28:28.730 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:28.730 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:28.730 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:28:28.731 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:28.731 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:28:28.731 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:28.731 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:28.731 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:28:28.732 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:28.732 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:28:28.732 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:28.732 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:28:28.732 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:28.733 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:28.733 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:28.733 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:28.733 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:28:28.733 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:28.734 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:28.734 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:28:28.734 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:28.734 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:28.735 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:28.735 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:28:28.735 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:28.735 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:28.735 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:28.736 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:28.736 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:28:28.736 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:28.736 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:28.736 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:28.737 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:28.737 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:28.737 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:28.737 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:28.737 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:28.738 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:28.738 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:28.738 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:28.738 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:28:28.739 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:28.739 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:28.739 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:28.739 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:28.739 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:28.740 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:28.740 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:28.740 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:28.740 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:28.740 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:28.741 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:28:28.741 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:28.741 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:28:28.741 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:28:28.741 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:28:28.742 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:28.742 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:28:28.742 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:28:33.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:33.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:33.745 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:28:33.902 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:28:33.902 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:28:34.018 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:28:34.019 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:28:34.019 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:28:34.019 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:28:34.019 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:28:34.019 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:28:34.020 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:34.020 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:34.020 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:28:34.020 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:34.021 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:28:34.021 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:34.021 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:34.021 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:28:34.021 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:34.022 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:28:34.022 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:34.022 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:28:34.022 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:34.022 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:34.023 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:34.023 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:34.023 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:28:34.023 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:34.024 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:34.024 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:28:34.024 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:34.024 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:34.025 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:34.025 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:28:34.025 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:34.025 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:34.025 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:34.026 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:34.026 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:28:34.026 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:34.026 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:34.027 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:34.027 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:34.027 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:34.027 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:34.028 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:34.028 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:34.028 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:34.028 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:34.028 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:34.029 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:28:34.029 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:34.029 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:34.029 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:34.029 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:34.030 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:34.030 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:34.030 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:34.030 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:34.030 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:34.031 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:34.031 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:28:34.031 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:34.031 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:28:34.032 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:28:34.032 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:28:34.032 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:34.032 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:28:34.033 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.946914_319.1 2021-09-17T10:28:39.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:39.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:39.036 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:28:39.077 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:28:39.078 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:28:39.209 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:28:39.210 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:28:39.210 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:28:39.211 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:28:39.211 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:28:39.212 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:28:39.212 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:39.213 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:39.213 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:28:39.213 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:39.214 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:28:39.215 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:39.215 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:39.216 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:28:39.216 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:39.217 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:28:39.217 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:39.218 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:28:39.218 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:39.219 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:39.219 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:39.219 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:39.220 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:28:39.220 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:39.221 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:39.221 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:28:39.222 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:39.222 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:39.222 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:39.223 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:28:39.223 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:39.224 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:39.224 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:39.225 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:39.225 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:28:39.226 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:39.226 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:39.227 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:39.227 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:39.228 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:39.228 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:39.229 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:39.229 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:39.230 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:39.230 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:39.230 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:39.231 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:28:39.231 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:39.232 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:39.232 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:39.233 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:39.233 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:39.234 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:39.234 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:39.234 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:39.235 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:39.235 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:39.236 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:28:39.236 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:39.237 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:28:39.237 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:28:39.238 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:28:39.238 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:39.239 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:28:39.241 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.009924_325.1 2021-09-17T10:28:44.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:44.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:44.243 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:28:44.305 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:28:44.306 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:28:44.606 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:28:44.606 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:28:44.606 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:28:44.607 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:28:44.607 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:28:44.607 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:28:44.607 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:44.608 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:44.608 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:28:44.608 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:44.608 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:28:44.608 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:44.609 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:44.609 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:28:44.609 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:44.609 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:28:44.610 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:44.610 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:28:44.610 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:44.610 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:44.611 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:44.611 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:44.611 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:28:44.611 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:44.611 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:44.612 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.256150_335.1", 2021-09-17T10:28:44.612 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:44.612 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:44.612 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:44.613 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.256150Z", 2021-09-17T10:28:44.613 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:44.613 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:44.613 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:44.613 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:44.616 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:44.616 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:44.616 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:44.616 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:28:44.617 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:44.617 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:44.617 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:44.617 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:44.617 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:44.618 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:44.618 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:44.618 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:44.618 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:44.619 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:44.619 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:28:44.619 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:44.619 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:28:44.619 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:28:44.620 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:28:44.620 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:44.620 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:28:44.620 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.256150_335.1 2021-09-17T10:28:49.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:49.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:49.625 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:28:49.905 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:28:49.905 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:28:50.025 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:28:50.025 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:28:50.026 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:28:50.026 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:28:50.027 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:28:50.027 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:28:50.028 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:50.028 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:50.029 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:28:50.029 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:50.030 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:28:50.030 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:50.031 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:50.031 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:28:50.031 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:50.032 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:28:50.033 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:50.033 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:28:50.033 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:50.033 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:50.034 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:50.034 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:50.034 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:28:50.034 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:50.035 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:50.035 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:28:50.035 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:50.035 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:50.035 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:50.036 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:28:50.036 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:50.036 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:50.036 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:50.036 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:50.037 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:28:50.037 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:50.037 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:50.037 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:50.038 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:50.038 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:50.038 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:50.038 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:50.038 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:50.039 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:50.039 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:50.039 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:50.039 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:28:50.039 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:50.040 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:50.040 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:50.040 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:50.040 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:50.041 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:50.041 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:50.041 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:50.041 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:50.041 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:50.042 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:28:50.042 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:50.042 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:28:50.042 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:28:50.043 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:28:50.043 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:50.043 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:28:50.045 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.009924_325.1 2021-09-17T10:28:55.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:55.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:28:55.047 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:28:55.162 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:28:55.162 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:28:55.542 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:28:55.542 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:28:55.542 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:28:55.542 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:28:55.543 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:28:55.543 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:28:55.543 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:55.544 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:55.544 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:28:55.544 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:55.544 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:28:55.545 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:55.545 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:55.545 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:28:55.546 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:55.546 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:28:55.547 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:55.547 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:28:55.547 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:55.547 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:55.548 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:55.548 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:55.548 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:28:55.548 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:55.549 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:55.549 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:28:55.549 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:55.549 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:55.550 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:55.550 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:28:55.550 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:55.550 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:55.550 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:55.551 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:55.551 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:28:55.551 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:55.551 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:55.552 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:55.552 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:55.552 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:55.552 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:55.553 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:55.553 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:55.553 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:55.553 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:55.553 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:28:55.554 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:28:55.554 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:28:55.554 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:28:55.554 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:28:55.555 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:28:55.555 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:28:55.555 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:28:55.555 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:28:55.556 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:28:55.556 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:55.556 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:55.556 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:28:55.557 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:28:55.557 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:28:55.557 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:28:55.557 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:28:55.557 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:28:55.558 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:28:55.559 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:29:00.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:00.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:00.562 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:29:00.606 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:29:00.606 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:29:00.730 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:29:00.730 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:29:00.731 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:29:00.731 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:29:00.731 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:29:00.731 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:29:00.732 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:00.732 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:00.732 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:29:00.732 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:00.733 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:29:00.733 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:00.733 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:00.733 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:29:00.734 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:00.734 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:29:00.734 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:00.734 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:29:00.734 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:00.735 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:00.735 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:00.735 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:00.735 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:29:00.736 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:00.736 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:00.736 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:29:00.736 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:00.737 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:00.737 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:00.737 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:29:00.737 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:00.738 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:00.738 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:00.738 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:00.738 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:29:00.739 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:00.739 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:00.739 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:00.739 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:00.740 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:00.740 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:00.740 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:00.740 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:00.741 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:00.741 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:00.741 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:00.741 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:29:00.742 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:00.742 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:00.742 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:00.742 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:00.742 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:00.743 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:00.743 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:00.743 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:00.743 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:00.744 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:00.744 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:29:00.744 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:00.744 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:29:00.744 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:29:00.745 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:29:00.745 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:00.745 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:29:00.746 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:29:05.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:05.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:05.749 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:29:05.807 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:29:05.807 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:29:06.256 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:29:06.257 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:29:06.257 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:29:06.257 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:29:06.258 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:29:06.258 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:29:06.258 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:06.258 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:06.259 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:29:06.259 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:06.259 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:29:06.259 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:06.260 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:06.260 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:29:06.260 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:06.260 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:29:06.261 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:06.261 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:29:06.261 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:06.261 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:06.261 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:06.262 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:06.262 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:29:06.262 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:06.262 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:06.263 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.256150_335.1", 2021-09-17T10:29:06.263 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:06.263 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:06.263 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:06.263 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.256150Z", 2021-09-17T10:29:06.264 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:06.264 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:06.264 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:06.265 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:06.265 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:29:06.265 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:06.265 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:06.266 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:06.266 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:06.266 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:06.266 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:06.267 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:06.267 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:06.267 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:06.267 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:06.267 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:06.268 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:29:06.268 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:06.268 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:06.268 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:06.269 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:06.269 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:06.269 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:06.269 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:06.270 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:06.270 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:06.270 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:06.270 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:29:06.270 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:06.271 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:29:06.271 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:29:06.271 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:29:06.271 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:06.272 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:29:06.273 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.256150_335.1 2021-09-17T10:29:11.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:11.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:11.275 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:29:11.317 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:29:11.317 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:29:11.936 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:29:11.936 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:29:11.937 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:29:11.937 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:29:11.937 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:29:11.937 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:29:11.938 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:11.938 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:11.938 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:29:11.939 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:11.939 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:29:11.939 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:11.939 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:11.940 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:29:11.940 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:11.940 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:29:11.940 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:11.940 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:29:11.941 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:11.941 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:11.941 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:11.941 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:11.942 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:29:11.942 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:11.942 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:11.942 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:29:11.943 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:11.943 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:11.943 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:11.943 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:29:11.944 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:11.944 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:11.944 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:11.944 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:11.945 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:29:11.945 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:11.945 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:11.946 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:11.946 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:11.946 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:11.946 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:11.947 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:11.947 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:11.947 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:11.947 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:11.947 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:11.948 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:29:11.948 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:11.948 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:11.948 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:11.949 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:11.949 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:11.949 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:11.949 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:11.950 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:11.950 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:11.950 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:11.950 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:29:11.951 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:11.951 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:29:11.951 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:29:11.951 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:29:11.952 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:11.952 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:29:11.953 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.009924_325.1 2021-09-17T10:29:16.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:16.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:16.956 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:29:17.038 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:29:17.038 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:29:17.423 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:29:17.423 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:29:17.423 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:29:17.424 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:29:17.424 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:29:17.424 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:29:17.424 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:17.425 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:17.425 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:29:17.425 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:17.425 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:29:17.425 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:17.426 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:17.426 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:29:17.426 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:17.426 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:29:17.426 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:17.427 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:29:17.427 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:17.427 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:17.427 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:17.427 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:17.428 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:29:17.428 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:17.428 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:17.428 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:29:17.428 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:17.429 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:17.429 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:17.429 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:29:17.429 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:17.430 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:17.430 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:17.430 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:17.430 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:29:17.430 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:17.431 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:17.431 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:17.431 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:17.431 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:17.431 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:17.432 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:17.432 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:17.432 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:17.432 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:17.432 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:17.433 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:29:17.433 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:17.433 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:17.433 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:17.433 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:17.434 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:17.434 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:17.434 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:17.434 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:17.435 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:17.435 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:17.435 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:29:17.435 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:17.435 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:29:17.436 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:29:17.436 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:29:17.436 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:17.436 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:29:17.437 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.009924_325.1 2021-09-17T10:29:22.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:22.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:22.440 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:29:22.481 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:29:22.481 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:29:22.601 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:29:22.601 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:29:22.602 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:29:22.602 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:29:22.602 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:29:22.603 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:29:22.603 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:22.603 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:22.603 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:29:22.603 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:22.604 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:29:22.604 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:22.604 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:22.604 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:29:22.604 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:22.605 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:29:22.605 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:22.605 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:29:22.605 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:22.606 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:22.606 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:22.606 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:22.606 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:29:22.607 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:22.607 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:22.607 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:29:22.607 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:22.607 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:22.608 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:22.608 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:29:22.608 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:22.608 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:22.609 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:22.609 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:22.609 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:29:22.609 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:22.609 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:22.610 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:22.610 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:22.610 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:22.610 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:22.611 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:22.611 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:22.611 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:22.611 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:22.611 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:22.612 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:29:22.612 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:22.612 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:22.612 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:22.612 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:22.613 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:22.613 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:22.613 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:22.613 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:22.614 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:22.614 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:22.614 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:29:22.614 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:22.615 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:29:22.615 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:29:22.615 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:29:22.615 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:22.615 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:29:22.617 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:29:27.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:27.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:27.620 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:29:27.662 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:29:27.662 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:29:27.877 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:29:27.878 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:29:27.878 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:29:27.878 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:29:27.878 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:29:27.879 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:29:27.879 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:27.879 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:27.879 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:29:27.880 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:27.880 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:29:27.880 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:27.880 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:27.880 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:29:27.881 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:27.881 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:29:27.881 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:27.882 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:29:27.882 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:27.882 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:27.882 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:27.882 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:27.883 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:29:27.883 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:27.883 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:27.883 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:29:27.884 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:27.884 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:27.884 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:27.884 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:29:27.884 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:27.885 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:27.885 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:27.885 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:27.885 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:29:27.886 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:27.886 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:27.886 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:27.886 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:27.887 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:27.887 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:27.888 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:27.888 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:27.888 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:27.889 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:27.889 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:27.889 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:29:27.889 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:27.890 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:27.890 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:27.890 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:27.890 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:27.891 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:27.891 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:27.892 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:27.892 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:27.892 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:27.893 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:29:27.893 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:27.893 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:29:27.894 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:29:27.894 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:29:27.895 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:27.895 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:29:27.896 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.952206_320.1 2021-09-17T10:29:32.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:32.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:32.899 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:29:32.960 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:29:32.960 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:29:33.214 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:29:33.214 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:29:33.215 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:29:33.215 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:29:33.215 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:29:33.215 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:29:33.216 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:33.216 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:33.216 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:29:33.216 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:33.216 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:29:33.217 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:33.217 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:33.217 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:29:33.217 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:33.218 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:29:33.218 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:33.218 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:29:33.218 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:33.219 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:33.219 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:33.219 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:33.219 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:29:33.219 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:33.220 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:33.220 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.164606_332.1", 2021-09-17T10:29:33.220 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:33.220 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:33.221 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:33.221 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.164606Z", 2021-09-17T10:29:33.221 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:33.221 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:33.221 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:33.222 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:33.222 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:29:33.222 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:33.222 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:33.223 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:33.223 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:33.223 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:33.223 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:33.224 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:33.224 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:33.224 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:33.224 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:33.224 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:33.225 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:29:33.225 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:33.225 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:33.225 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:33.226 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:33.226 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:33.226 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:33.226 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:33.226 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:33.227 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:33.227 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:33.227 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:29:33.227 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:33.228 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:29:33.228 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:29:33.228 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:29:33.228 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:33.228 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:29:33.230 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.164606_332.1 2021-09-17T10:29:38.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:38.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:38.233 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:29:38.273 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:29:38.274 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:29:38.393 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:29:38.393 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:29:38.393 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:29:38.394 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:29:38.394 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:29:38.394 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:29:38.394 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:38.394 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:38.395 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:29:38.395 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:38.395 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:29:38.395 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:38.396 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:38.396 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:29:38.396 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:38.396 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:29:38.397 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:38.397 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:29:38.397 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:38.397 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:38.397 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:38.398 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:38.398 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:29:38.398 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:38.398 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:38.399 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:29:38.399 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:38.399 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:38.399 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:38.400 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:29:38.400 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:38.400 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:38.400 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:38.400 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:38.401 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:29:38.401 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:38.401 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:38.401 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:38.402 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:38.402 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:38.402 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:38.402 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:38.402 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:38.403 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:38.403 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:38.403 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:38.403 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:29:38.404 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:38.404 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:38.404 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:38.404 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:38.405 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:38.405 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:38.405 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:38.405 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:38.405 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:38.406 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:38.406 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:29:38.406 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:38.406 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:29:38.407 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:29:38.407 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:29:38.407 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:38.407 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:29:38.408 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:29:43.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:43.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:43.411 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:29:43.559 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:29:43.560 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:29:43.955 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:29:43.956 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:29:43.956 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:29:43.956 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:29:43.957 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:29:43.957 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:29:43.957 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:43.958 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:43.958 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:29:43.958 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:43.959 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:29:43.959 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:43.959 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:43.959 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:29:43.960 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:43.960 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:29:43.960 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:43.960 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:29:43.961 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:43.961 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:43.961 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:43.961 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:43.961 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:29:43.962 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:43.962 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:43.962 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:29:43.962 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:43.963 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:43.963 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:43.963 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:29:43.963 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:43.963 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:43.964 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:43.964 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:43.964 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:29:43.964 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:43.965 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:43.965 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:43.965 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:43.965 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:43.965 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:43.966 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:43.966 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:43.966 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:43.966 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:43.967 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:43.967 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:29:43.967 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:43.967 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:43.968 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:43.968 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:43.968 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:43.968 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:43.969 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:43.969 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:43.969 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:43.969 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:43.969 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:29:43.970 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:43.970 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:29:43.970 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:29:43.970 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:29:43.971 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:43.971 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:29:43.972 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.009924_325.1 2021-09-17T10:29:48.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:48.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:48.975 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:29:49.015 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:29:49.016 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:29:49.118 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:29:49.119 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:29:49.119 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:29:49.119 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:29:49.119 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:29:49.120 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:29:49.120 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:49.120 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:49.120 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:29:49.121 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:49.121 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:29:49.121 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:49.121 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:49.121 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:29:49.122 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:49.122 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:29:49.122 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:49.122 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:29:49.123 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:49.123 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:49.123 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:49.123 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:49.123 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:29:49.124 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:49.124 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:49.124 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.946914_319.1", 2021-09-17T10:29:49.124 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:49.125 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:49.125 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:49.125 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.946914Z", 2021-09-17T10:29:49.125 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:49.126 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:49.126 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:49.126 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:49.126 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:29:49.126 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:49.127 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:49.127 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:49.127 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:49.127 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:49.128 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:49.128 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:49.128 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:49.128 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:49.128 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:49.129 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:49.129 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:29:49.129 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:49.129 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:49.129 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:49.130 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:49.130 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:49.130 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:49.130 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:49.131 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:49.131 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:49.131 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:49.131 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:29:49.131 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:49.132 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:29:49.132 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:29:49.132 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:29:49.132 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:49.132 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:29:49.134 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.946914_319.1 2021-09-17T10:29:54.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:54.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:54.136 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:29:54.177 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:29:54.177 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:29:54.290 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:29:54.291 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:29:54.291 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:29:54.291 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:29:54.292 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:29:54.292 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:29:54.292 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:54.292 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:54.293 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:29:54.293 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:54.293 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:29:54.294 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:54.294 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:54.294 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:29:54.294 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:54.295 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:29:54.295 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:54.295 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:29:54.295 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:54.296 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:54.296 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:54.296 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:54.296 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:29:54.296 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:54.297 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:54.297 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.007158_324.1", 2021-09-17T10:29:54.297 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:54.297 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:54.298 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:54.298 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.007158Z", 2021-09-17T10:29:54.298 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:54.298 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:54.298 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:54.299 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:54.299 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:29:54.299 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:54.299 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:54.300 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:54.300 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:54.300 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:54.300 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:54.300 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:54.301 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:54.301 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:54.301 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:54.301 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:54.302 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:29:54.302 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:54.302 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:54.303 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:54.303 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:54.303 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:54.304 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:54.304 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:54.305 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:54.305 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:54.306 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:54.306 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:29:54.306 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:54.306 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:29:54.307 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:29:54.307 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:29:54.307 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:54.307 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:29:54.309 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.007158_324.1 2021-09-17T10:29:59.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:59.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:29:59.312 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:29:59.373 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:29:59.738 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:29:59.738 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:29:59.739 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:29:59.739 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:29:59.739 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:29:59.739 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:29:59.740 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:59.740 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:59.740 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:29:59.741 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:59.741 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:29:59.741 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:59.741 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:59.742 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:29:59.742 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:59.742 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:29:59.742 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:59.743 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:29:59.743 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:59.743 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:59.743 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:59.744 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:59.744 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:29:59.744 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:59.744 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:59.744 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:29:59.745 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:59.745 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:59.745 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:59.746 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:29:59.746 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:59.746 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:59.746 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:59.747 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:59.747 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:29:59.747 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:59.747 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:59.748 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:59.748 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:59.748 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:59.748 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:59.749 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:59.749 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:59.749 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:59.749 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:59.750 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:29:59.750 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:29:59.750 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:29:59.750 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:29:59.751 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:29:59.751 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:29:59.751 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:29:59.751 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:29:59.752 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:29:59.752 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:29:59.752 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:59.752 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:59.752 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:29:59.753 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:29:59.753 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:29:59.753 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:29:59.753 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:29:59.754 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:29:59.754 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:29:59.755 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.954369_321.1 2021-09-17T10:30:04.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:30:04.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:30:04.758 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:30:04.799 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:30:04.800 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:30:04.921 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:30:04.922 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:30:04.922 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:30:04.922 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:30:04.923 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:30:04.923 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:30:04.923 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:04.923 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:04.924 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:30:04.924 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:04.924 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:30:04.924 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:04.925 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:04.925 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:30:04.925 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:04.926 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:30:04.926 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:04.926 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:30:04.926 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:04.927 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:04.927 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:04.927 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:04.927 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:30:04.928 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:04.928 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:04.928 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.105095_330.1", 2021-09-17T10:30:04.928 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:04.929 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:04.929 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:04.929 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.105095Z", 2021-09-17T10:30:04.929 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:04.929 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:04.930 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:04.930 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:04.930 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:30:04.930 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:04.931 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:04.931 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:30:04.931 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:04.931 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:04.932 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:04.932 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:30:04.932 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:04.932 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:04.933 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:04.933 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:04.933 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:30:04.933 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:04.933 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:04.934 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:30:04.934 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:04.934 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:04.934 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:04.935 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:30:04.935 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:04.935 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:04.936 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:04.936 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:30:04.936 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:04.936 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:30:04.937 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:30:04.937 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:30:04.937 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:04.937 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:30:04.939 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.105095_330.1 2021-09-17T10:30:09.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:30:09.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:30:09.946 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:30:10.042 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:30:10.042 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:30:10.456 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:30:10.456 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:30:10.456 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:30:10.457 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:30:10.457 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:30:10.457 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:30:10.458 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:10.458 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:10.458 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:30:10.458 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:10.459 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:30:10.459 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:10.459 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:10.460 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:30:10.460 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:10.460 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:30:10.460 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:10.461 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:30:10.461 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:10.461 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:10.461 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:10.462 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:10.462 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:30:10.462 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:10.462 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:10.463 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.007158_324.1", 2021-09-17T10:30:10.463 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:10.463 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:10.464 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:10.464 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.007158Z", 2021-09-17T10:30:10.465 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:10.465 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:10.466 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:10.466 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:10.467 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:30:10.467 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:10.468 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:10.468 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:30:10.469 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:10.469 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:10.469 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:10.470 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:30:10.470 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:10.471 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:10.471 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:10.472 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:10.472 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:30:10.473 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:10.473 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:10.474 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:30:10.474 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:10.474 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:10.475 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:10.475 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:30:10.476 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:10.476 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:10.476 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:10.477 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:30:10.477 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:10.478 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:30:10.478 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:30:10.479 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:30:10.479 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:10.479 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:30:10.481 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.007158_324.1 2021-09-17T10:30:15.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:30:15.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:30:15.485 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:30:15.540 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:30:15.540 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:30:15.624 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:30:15.624 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:30:15.625 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:30:15.625 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:30:15.625 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:30:15.626 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:30:15.626 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:15.626 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:15.626 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:30:15.627 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:15.627 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:30:15.627 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:15.627 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:15.628 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:30:15.628 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:15.628 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:30:15.628 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:15.629 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:30:15.629 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:15.629 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:15.629 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:15.630 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:15.630 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:30:15.630 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:15.630 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:15.631 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:30:15.631 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:15.631 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:15.631 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:15.632 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:30:15.632 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:15.632 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:15.632 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:15.633 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:15.633 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:30:15.633 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:15.634 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:15.634 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:30:15.634 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:15.634 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:15.635 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:15.635 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:30:15.635 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:15.635 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:15.636 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:15.636 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:15.636 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:30:15.636 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:15.637 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:15.637 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:30:15.637 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:15.637 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:15.638 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:15.638 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:30:15.638 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:15.638 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:15.639 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:15.639 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:30:15.639 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:15.639 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:30:15.640 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:30:15.640 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:30:15.640 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:15.640 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:30:15.642 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873653.009924_325.1 2021-09-17T10:30:20.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:30:20.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:30:20.644 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:30:20.685 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:30:20.686 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:30:20.801 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:30:20.802 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:30:20.802 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:30:20.802 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:30:20.803 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:30:20.803 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:30:20.803 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:20.803 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:20.804 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:30:20.804 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:20.804 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:30:20.804 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:20.804 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:20.805 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:30:20.805 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:20.805 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:30:20.805 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:20.806 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:30:20.806 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:20.806 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:20.806 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:20.806 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:20.807 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:30:20.807 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:20.807 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:20.807 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.922243_318.1", 2021-09-17T10:30:20.808 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:20.808 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:20.808 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:20.808 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.922243Z", 2021-09-17T10:30:20.808 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:20.809 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:20.809 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:20.809 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:20.809 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:30:20.809 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:20.810 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:20.810 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:30:20.810 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:20.810 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:20.811 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:20.811 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:30:20.811 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:20.811 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:20.811 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:20.812 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:20.812 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:30:20.812 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:20.812 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:20.812 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:30:20.813 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:20.813 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:20.813 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:20.813 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:30:20.813 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:20.814 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:20.814 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:20.814 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:30:20.814 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:20.815 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:30:20.815 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:30:20.815 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:30:20.815 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:20.815 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:30:20.817 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.922243_318.1 2021-09-17T10:30:25.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:30:25.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:30:25.820 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:30:25.857 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:30:25.857 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:30:26.279 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:30:26.279 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:30:26.279 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:30:26.279 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:30:26.280 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:30:26.280 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:30:26.280 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:26.280 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:26.281 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:30:26.281 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:26.281 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:30:26.281 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:26.282 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:26.282 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:30:26.282 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:26.282 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:30:26.283 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:26.283 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:30:26.283 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:26.283 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:26.283 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:26.284 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:26.284 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:30:26.284 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:26.284 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:26.284 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:30:26.285 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:26.285 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:26.285 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:26.285 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:30:26.286 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:26.286 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:26.286 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:26.286 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:26.287 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:30:26.287 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:26.288 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:26.288 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:30:26.288 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:26.288 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:26.289 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:26.289 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:30:26.289 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:26.289 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:26.289 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:26.290 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:30:26.290 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:30:26.290 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:30:26.290 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:30:26.291 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:30:26.291 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:30:26.291 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:30:26.291 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:30:26.292 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:30:26.292 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:30:26.292 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:26.292 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:26.292 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:30:26.293 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:30:26.293 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:30:26.293 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:30:26.293 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:30:26.293 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:30:26.294 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:30:26.294 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874310.992181_435.1 target=1_1631873652.952206_320.1 2021-09-17T10:30:31.296 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2021-09-17T10:30:31.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:30:31.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:30:31.299 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T10:30:31.342 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:30:31.343 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:30:31.395 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:30:31.416+0000 7fad04519b00 0 failed reading obj info from .rgw.root:zone_info.9854c5a3-6cc0-45a7-b530-3af2b1c733d5: (2) No such file or directory 2021-09-17T10:30:31.395 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:30:31.416+0000 7fad04519b00 0 WARNING: could not read zone params for zone id=9854c5a3-6cc0-45a7-b530-3af2b1c733d5 name=test-zone1 2021-09-17T10:30:31.395 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:30:31.416+0000 7fad04519b00 0 failed reading obj info from .rgw.root:zone_info.d9facd38-bb4e-4ea2-9817-2655cc5c6379: (2) No such file or directory 2021-09-17T10:30:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:2021-09-17T10:30:31.416+0000 7fad04519b00 0 WARNING: could not read zone params for zone id=d9facd38-bb4e-4ea2-9817-2655cc5c6379 name=test-zone3 2021-09-17T10:30:31.403 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:30:31.403 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:30:31.404 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone2", 2021-09-17T10:30:31.404 INFO:teuthology.orchestra.run.smithi050.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-17T10:30:31.404 INFO:teuthology.orchestra.run.smithi050.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-17T10:30:31.404 INFO:teuthology.orchestra.run.smithi050.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-17T10:30:31.405 INFO:teuthology.orchestra.run.smithi050.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-17T10:30:31.405 INFO:teuthology.orchestra.run.smithi050.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-17T10:30:31.405 INFO:teuthology.orchestra.run.smithi050.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-17T10:30:31.405 INFO:teuthology.orchestra.run.smithi050.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-17T10:30:31.406 INFO:teuthology.orchestra.run.smithi050.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-17T10:30:31.406 INFO:teuthology.orchestra.run.smithi050.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-17T10:30:31.406 INFO:teuthology.orchestra.run.smithi050.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-17T10:30:31.406 INFO:teuthology.orchestra.run.smithi050.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-17T10:30:31.407 INFO:teuthology.orchestra.run.smithi050.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-17T10:30:31.407 INFO:teuthology.orchestra.run.smithi050.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-17T10:30:31.407 INFO:teuthology.orchestra.run.smithi050.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-17T10:30:31.407 INFO:teuthology.orchestra.run.smithi050.stdout: "system_key": { 2021-09-17T10:30:31.408 INFO:teuthology.orchestra.run.smithi050.stdout: "access_key": "H92RM4Y9KBWIE4BC", 2021-09-17T10:30:31.408 INFO:teuthology.orchestra.run.smithi050.stdout: "secret_key": "RszMO0lPlmVPzLDGoCXqxRAVf96gVQmK" 2021-09-17T10:30:31.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:30:31.408 INFO:teuthology.orchestra.run.smithi050.stdout: "placement_pools": [ 2021-09-17T10:30:31.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:30:31.409 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "default-placement", 2021-09-17T10:30:31.409 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:30:31.409 INFO:teuthology.orchestra.run.smithi050.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-17T10:30:31.410 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_classes": { 2021-09-17T10:30:31.410 INFO:teuthology.orchestra.run.smithi050.stdout: "STANDARD": { 2021-09-17T10:30:31.410 INFO:teuthology.orchestra.run.smithi050.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2021-09-17T10:30:31.410 INFO:teuthology.orchestra.run.smithi050.stdout: "compression_type": "random" 2021-09-17T10:30:31.411 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:30:31.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:30:31.411 INFO:teuthology.orchestra.run.smithi050.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-17T10:30:31.411 INFO:teuthology.orchestra.run.smithi050.stdout: "index_type": 0 2021-09-17T10:30:31.412 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:30:31.412 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:30:31.412 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:30:31.412 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:30:31.413 INFO:teuthology.orchestra.run.smithi050.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-17T10:30:31.413 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:30:31.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:30:31.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:30:31.416 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-17T10:30:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:30:31.453 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:30:31.577 INFO:teuthology.orchestra.run.smithi050.stderr:Sending period to new master zone 9854c5a3-6cc0-45a7-b530-3af2b1c733d5 2021-09-17T10:30:31.636 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:30:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:30:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 5, 2021-09-17T10:30:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "predecessor_uuid": "20a9da20-85d6-489f-bcfe-ebc80031614a", 2021-09-17T10:30:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": [], 2021-09-17T10:30:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "period_map": { 2021-09-17T10:30:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:30:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "zonegroups": [ 2021-09-17T10:30:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:30:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "test-zonegroup", 2021-09-17T10:30:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zonegroup", 2021-09-17T10:30:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "api_name": "", 2021-09-17T10:30:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "is_master": "true", 2021-09-17T10:30:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:30:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:30:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:30:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: "hostnames": [], 2021-09-17T10:30:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: "hostnames_s3website": [], 2021-09-17T10:30:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:30:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "zones": [ 2021-09-17T10:30:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:30:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:30:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone2", 2021-09-17T10:30:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:30:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi050.front.sepia.ceph.com:8000" 2021-09-17T10:30:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:30:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "log_meta": "false", 2021-09-17T10:30:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "log_data": "true", 2021-09-17T10:30:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:30:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "read_only": "false", 2021-09-17T10:30:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_type": "", 2021-09-17T10:30:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from_all": "false", 2021-09-17T10:30:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from": [], 2021-09-17T10:30:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_zone": "" 2021-09-17T10:30:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:30:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:30:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "74241c6a-1297-4f4a-8bbd-b65c661636d9", 2021-09-17T10:30:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone4", 2021-09-17T10:30:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:30:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi050.front.sepia.ceph.com:8001" 2021-09-17T10:30:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:30:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "log_meta": "false", 2021-09-17T10:30:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "log_data": "true", 2021-09-17T10:30:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:30:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "read_only": "false", 2021-09-17T10:30:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_type": "pubsub", 2021-09-17T10:30:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from_all": "false", 2021-09-17T10:30:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from": [ 2021-09-17T10:30:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "test-zone1" 2021-09-17T10:30:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:30:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_zone": "" 2021-09-17T10:30:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:30:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:30:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:30:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone1", 2021-09-17T10:30:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:30:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:30:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:30:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "log_meta": "false", 2021-09-17T10:30:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "log_data": "true", 2021-09-17T10:30:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:30:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "read_only": "false", 2021-09-17T10:30:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_type": "", 2021-09-17T10:30:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from_all": "true", 2021-09-17T10:30:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from": [], 2021-09-17T10:30:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_zone": "" 2021-09-17T10:30:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:30:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:30:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "id": "d9facd38-bb4e-4ea2-9817-2655cc5c6379", 2021-09-17T10:30:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "test-zone3", 2021-09-17T10:30:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "endpoints": [ 2021-09-17T10:30:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "http://smithi018.front.sepia.ceph.com:8001" 2021-09-17T10:30:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:30:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "log_meta": "false", 2021-09-17T10:30:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "log_data": "true", 2021-09-17T10:30:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:30:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "read_only": "false", 2021-09-17T10:30:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_type": "", 2021-09-17T10:30:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from_all": "true", 2021-09-17T10:30:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_from": [], 2021-09-17T10:30:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "redirect_zone": "" 2021-09-17T10:30:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:30:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:30:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "placement_targets": [ 2021-09-17T10:30:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:30:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "default-placement", 2021-09-17T10:30:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "tags": [], 2021-09-17T10:30:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "storage_classes": [ 2021-09-17T10:30:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "STANDARD" 2021-09-17T10:30:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:30:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:30:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:30:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "default_placement": "default-placement", 2021-09-17T10:30:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:30:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_policy": { 2021-09-17T10:30:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "groups": [] 2021-09-17T10:30:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:30:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:30:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2021-09-17T10:30:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "short_zone_ids": [ 2021-09-17T10:30:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:30:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:30:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 1815852058 2021-09-17T10:30:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:30:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:30:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "74241c6a-1297-4f4a-8bbd-b65c661636d9", 2021-09-17T10:30:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 1495243106 2021-09-17T10:30:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:30:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:30:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:30:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 664423460 2021-09-17T10:30:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:30:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:30:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "d9facd38-bb4e-4ea2-9817-2655cc5c6379", 2021-09-17T10:30:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "val": 2096482420 2021-09-17T10:30:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:30:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:30:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:30:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T10:30:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:30:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "period_config": { 2021-09-17T10:30:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "bucket_quota": { 2021-09-17T10:30:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": false, 2021-09-17T10:30:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "check_on_raw": false, 2021-09-17T10:30:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size": -1, 2021-09-17T10:30:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size_kb": 0, 2021-09-17T10:30:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "max_objects": -1 2021-09-17T10:30:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:30:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "user_quota": { 2021-09-17T10:30:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "enabled": false, 2021-09-17T10:30:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "check_on_raw": false, 2021-09-17T10:30:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size": -1, 2021-09-17T10:30:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "max_size_kb": 0, 2021-09-17T10:30:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "max_objects": -1 2021-09-17T10:30:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:30:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:30:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:30:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_name": "test-realm", 2021-09-17T10:30:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:30:31.666 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:30:31.668 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-09-17T10:31:31.669 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-45 2021-09-17T10:31:31.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:31.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:31.757 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:31:31.910 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:31:31.910 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:31:32.217 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:31:32.379 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:31:32.380 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:31:32.380 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:31:32.380 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:31:32.380 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:31:32.381 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:31:32.381 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631874691.729253_479.1", 2021-09-17T10:31:32.381 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:31:31.729253Z" 2021-09-17T10:31:32.381 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:31:32.382 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:31:32.382 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:31:32.382 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:31:32.382 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:31:32.383 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:31:32.383 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:31:32.383 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:31:32.383 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:31:32.384 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:31:32.385 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:31:32.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:32.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:32.387 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:31:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:31:32.432 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:31:32.553 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:31:32.554 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:31:32.554 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:31:32.554 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:31:32.554 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:31:32.555 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:31:32.555 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:32.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:32.555 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:31:32.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:32.556 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:31:32.556 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:32.556 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:32.557 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:31:32.557 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:32.557 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:31:32.557 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:32.558 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:31:32.558 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:32.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:32.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:32.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:32.559 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:31:32.559 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:32.559 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:32.559 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:31:32.560 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:32.560 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:32.560 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:32.560 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:31:32.561 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:32.561 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:32.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:32.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:32.561 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:31:32.562 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:32.562 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:32.562 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:31:32.563 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:32.563 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:32.563 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:32.563 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:31:32.563 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:32.564 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:32.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:32.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:32.564 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:31:32.565 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:32.565 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:32.565 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:31:32.565 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:32.566 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:32.566 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:32.566 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:31:32.566 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:32.566 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:32.567 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:32.567 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:31:32.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:32.567 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:31:32.568 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:31:32.568 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:31:32.568 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:32.568 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:31:32.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631874310.992181_435.1 2021-09-17T10:31:37.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:37.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:37.572 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:31:37.615 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:31:37.615 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:31:37.733 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:31:37.733 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:31:37.733 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:31:37.734 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:31:37.734 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:31:37.734 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:31:37.735 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:37.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:37.735 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:31:37.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:37.736 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:31:37.736 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:37.736 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:37.737 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:31:37.737 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:37.737 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:31:37.737 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:37.737 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:31:37.738 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:37.738 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:37.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:37.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:37.739 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:31:37.739 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:37.740 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:37.740 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:31:37.740 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:37.740 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:37.740 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:37.741 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:31:37.741 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:37.741 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:37.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:37.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:37.742 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:31:37.742 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:37.742 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:37.743 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:31:37.743 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:37.743 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:37.743 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:37.744 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:31:37.744 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:37.744 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:37.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:37.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:37.745 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:31:37.745 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:37.746 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:37.746 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:31:37.746 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:37.746 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:37.747 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:37.747 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:31:37.747 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:37.747 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:37.748 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:37.748 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:31:37.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:37.748 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:31:37.749 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:31:37.749 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:31:37.749 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:37.749 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:31:37.751 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631874310.992181_435.1 2021-09-17T10:31:42.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:42.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:42.754 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:31:42.796 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:31:42.797 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:31:42.920 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:31:42.921 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:31:42.921 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:31:42.921 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:31:42.921 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:31:42.922 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:31:42.922 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:42.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:42.922 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:31:42.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:42.923 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:31:42.923 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:42.923 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:42.924 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:31:42.924 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:42.924 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:31:42.924 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:42.924 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:31:42.925 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:42.925 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:42.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:42.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:42.926 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:31:42.926 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:42.926 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:42.926 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:31:42.926 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:42.927 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:42.927 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:42.927 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:31:42.927 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:42.928 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:42.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:42.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:42.928 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:31:42.928 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:31:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:31:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:31:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:31:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:31:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:31:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:31:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:31:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:31:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:42.935 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:31:42.936 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631874310.992181_435.1 2021-09-17T10:31:47.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:47.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:47.939 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:31:47.983 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:31:47.983 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:31:48.104 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:31:48.105 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:31:48.105 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:31:48.105 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:31:48.106 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:31:48.106 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:31:48.106 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:48.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:48.106 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:31:48.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:48.107 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:31:48.107 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:48.107 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:48.108 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:31:48.108 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:48.108 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:31:48.108 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:48.108 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:31:48.109 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:48.109 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:48.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:48.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:48.110 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:31:48.110 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:48.110 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:48.110 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:31:48.110 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:48.111 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:48.111 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:48.111 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:31:48.111 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:48.112 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:48.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:48.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:48.112 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:31:48.112 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:48.113 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:48.113 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:31:48.113 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:48.113 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:48.113 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:48.114 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:31:48.114 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:48.114 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:48.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:48.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:48.115 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:31:48.115 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:48.115 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:48.115 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:31:48.115 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:48.116 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:48.116 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:48.116 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:31:48.116 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:48.116 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:48.117 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:48.117 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:31:48.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:48.117 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:31:48.117 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:31:48.118 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:31:48.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:48.118 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:31:48.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631874310.992181_435.1 2021-09-17T10:31:53.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:53.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:53.122 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:31:53.164 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:31:53.164 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:31:53.284 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:31:53.284 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:31:53.284 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:31:53.285 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:31:53.285 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:31:53.285 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:31:53.285 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:53.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:53.286 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:31:53.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:53.286 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:31:53.286 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:53.287 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:53.287 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:31:53.287 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:53.287 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:31:53.287 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:53.288 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:31:53.288 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:53.288 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:53.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:53.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:53.289 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:31:53.289 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:53.289 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:53.289 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:31:53.290 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:53.290 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:53.290 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:53.290 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:31:53.291 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:53.291 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:53.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:53.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:53.291 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:31:53.292 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:53.292 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:53.292 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:31:53.292 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:53.293 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:53.293 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:53.293 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:31:53.293 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:53.293 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:53.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:53.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:53.294 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:31:53.294 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:53.294 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:53.295 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:31:53.295 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:53.295 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:53.295 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:53.296 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:31:53.296 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:53.296 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:53.296 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:53.296 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:31:53.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:53.297 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:31:53.297 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:31:53.297 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:31:53.297 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:53.298 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:31:53.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631874310.992181_435.1 2021-09-17T10:31:58.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:58.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:31:58.303 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:31:58.348 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:31:58.349 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:31:58.462 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:31:58.463 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:31:58.463 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:31:58.463 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:31:58.463 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:31:58.464 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:31:58.464 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:58.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:58.464 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:31:58.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:58.465 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:31:58.465 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:58.465 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:58.465 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:31:58.466 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:58.466 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:31:58.466 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:58.466 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:31:58.467 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:58.467 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:58.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:58.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:58.468 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:31:58.468 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:58.468 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:58.468 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:31:58.469 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:58.469 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:58.469 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:58.469 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:31:58.470 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:58.470 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:58.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:58.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:58.471 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:31:58.471 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:58.471 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:58.471 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:31:58.471 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:58.472 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:58.472 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:58.472 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:31:58.472 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:58.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:58.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:58.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:31:58.473 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:31:58.473 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:31:58.474 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:31:58.474 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:31:58.474 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:31:58.474 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:31:58.474 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:31:58.475 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:31:58.475 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:31:58.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:58.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:58.476 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:31:58.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:31:58.476 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:31:58.476 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:31:58.477 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:31:58.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:31:58.477 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:31:58.478 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631874310.992181_435.1 2021-09-17T10:32:01.835 INFO:tasks.rgw.c2.client.1.smithi050.stdout:2021-09-17T10:32:01.853+0000 7f0ae07f8700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-17T10:32:03.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:03.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:03.480 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:32:03.524 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:32:03.525 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:32:03.645 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:32:03.646 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:32:03.646 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:32:03.646 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:32:03.646 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:32:03.647 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:32:03.647 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:03.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:03.647 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:32:03.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:03.648 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:32:03.648 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:03.648 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:03.648 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:32:03.649 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:03.649 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:32:03.649 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:03.649 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:32:03.650 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:03.650 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:03.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:03.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:03.651 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:32:03.651 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:03.651 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:03.651 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:32:03.651 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:03.652 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:03.652 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:03.652 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:32:03.652 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:03.653 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:03.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:03.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:03.653 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:32:03.654 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:03.654 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:03.654 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:03.654 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:03.654 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:03.655 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:03.655 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:03.655 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:03.655 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:03.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:03.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:03.656 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:32:03.656 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:03.656 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:03.657 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:03.657 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:03.657 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:03.657 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:03.658 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:03.658 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:03.658 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:03.658 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:03.659 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:32:03.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:03.659 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:32:03.659 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:32:03.660 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:32:03.660 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:03.660 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:32:03.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631874310.992181_435.1 2021-09-17T10:32:08.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:08.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:08.665 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:32:08.707 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:32:08.707 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:32:08.828 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:32:08.828 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:32:08.829 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:32:08.829 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:32:08.829 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:32:08.830 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:32:08.830 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:08.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:08.830 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:32:08.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:08.831 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:32:08.831 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:08.831 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:08.831 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:32:08.832 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:08.832 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:32:08.832 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:08.832 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:32:08.832 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:08.833 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:08.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:08.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:08.833 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:32:08.834 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:08.834 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:08.834 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:32:08.834 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:08.834 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:08.835 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:08.835 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:32:08.835 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:08.835 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:08.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:08.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:08.836 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:32:08.836 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:08.837 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:08.837 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:08.837 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:08.837 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:08.837 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:08.838 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:08.838 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:08.838 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:08.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:08.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:08.839 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:32:08.839 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:08.839 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:08.839 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:08.840 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:08.840 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:08.840 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:08.840 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:08.840 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:08.841 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:08.841 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:08.841 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:32:08.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:08.842 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:32:08.842 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:32:08.842 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:32:08.842 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:08.842 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:32:08.844 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631874310.992181_435.1 2021-09-17T10:32:13.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:13.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:13.846 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:32:13.890 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:32:13.890 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:32:14.013 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:32:14.013 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:32:14.013 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:32:14.014 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:32:14.014 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:32:14.014 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:32:14.014 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:14.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:14.015 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:32:14.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:14.015 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:32:14.015 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:14.016 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:14.016 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:32:14.016 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:14.016 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:32:14.017 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:14.017 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:32:14.017 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:14.017 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:14.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:14.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:14.018 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:32:14.018 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:14.018 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:14.019 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:32:14.019 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:14.019 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:14.019 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:14.020 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:32:14.020 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:14.020 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:14.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:14.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:14.021 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:32:14.021 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:14.021 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:14.021 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:14.022 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:14.022 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:14.022 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:14.022 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:14.023 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:14.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:14.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:14.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:14.023 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:32:14.024 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:14.024 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:14.024 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:14.024 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:14.025 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:14.025 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:14.025 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:14.025 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:14.026 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:14.026 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:14.026 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:32:14.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:14.027 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:32:14.027 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:32:14.027 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:32:14.027 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:14.028 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:32:14.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631874310.992181_435.1 2021-09-17T10:32:19.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:19.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:19.032 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:32:19.073 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:32:19.073 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:32:19.193 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:32:19.193 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:32:19.193 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:32:19.194 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:32:19.194 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:32:19.194 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:32:19.194 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:19.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:19.195 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:32:19.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:19.195 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:32:19.196 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:19.196 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:19.196 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:32:19.196 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:19.196 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:32:19.197 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:19.197 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:32:19.197 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:19.197 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:19.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:19.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:19.199 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:32:19.199 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:19.199 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:19.200 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:32:19.200 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:19.200 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:19.200 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:19.200 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:32:19.201 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:19.201 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:19.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:19.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:19.202 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:32:19.202 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:19.202 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:19.202 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:19.203 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:19.203 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:19.203 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:19.203 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:19.203 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:19.204 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:19.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:19.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:19.204 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:32:19.205 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:19.205 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:19.205 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:19.205 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:19.205 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:19.206 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:19.206 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:19.206 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:19.206 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:32:19.208 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:32:19.208 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:19.208 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:32:19.209 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631874310.992181_435.1 2021-09-17T10:32:24.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:24.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:24.212 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:32:24.256 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:32:24.256 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:32:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:32:24.376 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:32:24.376 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:32:24.376 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:32:24.376 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:32:24.376 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:32:24.377 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:24.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:24.377 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:32:24.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:24.378 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:32:24.378 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:24.378 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:24.378 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:32:24.379 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:24.379 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:32:24.379 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:24.379 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:32:24.380 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:24.380 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:24.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:24.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:24.380 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:32:24.381 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:24.381 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:24.381 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:32:24.381 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:24.382 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:24.382 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:24.382 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:32:24.382 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:24.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:24.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:24.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:24.383 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:32:24.384 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:24.384 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:24.384 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:24.384 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:24.384 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:24.385 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:24.385 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:24.385 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:24.385 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:24.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:24.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:24.386 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:32:24.386 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:24.387 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:24.387 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:24.387 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:24.387 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:24.388 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:24.388 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:24.388 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:24.388 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:24.388 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:24.389 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:32:24.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:24.389 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:32:24.389 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:32:24.390 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:32:24.390 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:24.390 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:32:24.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631874310.992181_435.1 2021-09-17T10:32:29.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:29.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:29.393 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:32:29.562 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:32:29.563 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:32:29.563 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:32:29.563 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:32:29.563 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:32:29.564 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:32:29.564 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:29.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:29.565 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:32:29.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:29.565 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:32:29.565 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:29.565 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:29.566 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:32:29.566 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:29.566 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:32:29.566 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:29.567 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:32:29.567 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:29.567 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:29.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:29.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:29.568 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:32:29.568 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:29.568 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:29.569 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874310.992181_435.1", 2021-09-17T10:32:29.569 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:29.569 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:29.569 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:29.569 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:25:10.992181Z", 2021-09-17T10:32:29.570 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:29.570 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:29.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:29.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:29.571 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:32:29.571 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:29.571 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:29.571 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:29.571 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:29.572 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:29.572 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:29.572 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:29.572 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:29.572 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:29.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:29.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:29.573 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:32:29.573 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:29.574 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:29.574 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:29.574 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:29.574 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:29.575 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:29.575 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:29.575 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:29.575 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:29.575 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:29.576 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:32:29.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:29.576 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:32:29.576 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:32:29.577 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:32:29.577 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:29.577 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:32:29.578 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631874310.992181_435.1 2021-09-17T10:32:34.580 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:34.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:34.581 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:32:34.624 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:32:34.624 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:32:34.744 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:32:34.744 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:32:34.744 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:32:34.745 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:32:34.745 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:32:34.745 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:32:34.746 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:34.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:34.746 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:32:34.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:34.747 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:32:34.747 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:34.748 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:34.748 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:32:34.748 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:34.749 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:32:34.749 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:34.749 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:32:34.749 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:34.750 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:34.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:34.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:34.751 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:32:34.751 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:34.751 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:34.752 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874691.729253_479.1", 2021-09-17T10:32:34.752 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:34.752 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:34.753 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:34.753 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:31:31.729253Z", 2021-09-17T10:32:34.753 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:34.754 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:34.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:34.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:34.755 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:32:34.755 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:34.755 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:34.756 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:34.756 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:34.756 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:34.757 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:34.757 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:34.757 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:34.758 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:34.758 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:34.758 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:32:34.759 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:32:34.759 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:32:34.759 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:32:34.760 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:32:34.760 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:32:34.760 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:32:34.761 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:32:34.761 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:32:34.761 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:32:34.761 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:34.762 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:34.762 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:32:34.762 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:32:34.763 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:32:34.763 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:32:34.763 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:32:34.764 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:32:34.764 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:32:34.766 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:32:34.767 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:32:34.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:34.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:34.769 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:32:35.008 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:32:35.009 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:32:35.306 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:32:35.306 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:32:35.306 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:32:35.307 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:32:35.307 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:32:35.307 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:32:35.308 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:35.308 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:35.308 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:32:35.308 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:35.309 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:32:35.309 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:35.309 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:35.309 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:32:35.310 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:35.310 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:32:35.310 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:35.311 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:32:35.311 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:35.311 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:35.311 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:35.312 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:35.312 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:32:35.312 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:35.312 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:35.313 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:32:35.313 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:35.313 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:35.313 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:35.314 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:32:35.314 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:35.314 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:35.314 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:35.315 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:35.315 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:32:35.315 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:35.315 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:35.316 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:32:35.316 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:35.316 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:35.316 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:35.317 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:32:35.317 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:35.317 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:35.317 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:35.318 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:35.318 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:32:35.318 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:35.318 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:35.319 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:32:35.319 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:35.319 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:35.319 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:35.320 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:32:35.320 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:35.320 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:35.320 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:35.321 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:32:35.321 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:35.321 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:32:35.321 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:32:35.321 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:32:35.322 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:35.322 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:32:35.323 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:32:40.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:40.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:40.326 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:32:40.363 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:32:40.364 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:32:40.473 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:32:40.473 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:32:40.474 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:32:40.474 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:32:40.474 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:32:40.474 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:32:40.475 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:40.475 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:40.475 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:32:40.475 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:40.476 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:32:40.476 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:40.476 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:40.476 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:32:40.476 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:40.477 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:32:40.477 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:40.477 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:32:40.477 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:40.478 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:40.478 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:40.478 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:40.478 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:32:40.479 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:40.479 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:40.479 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:32:40.479 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:40.480 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:40.480 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:40.480 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:32:40.480 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:40.481 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:40.481 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:40.481 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:40.481 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:32:40.482 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:40.482 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:40.482 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:32:40.482 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:40.482 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:40.483 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:40.483 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:32:40.483 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:40.483 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:40.484 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:40.484 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:40.484 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:32:40.484 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:40.485 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:40.485 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:32:40.485 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:40.485 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:40.485 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:40.486 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:32:40.486 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:40.486 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:40.486 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:40.487 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:32:40.487 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:40.487 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:32:40.487 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:32:40.488 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:32:40.488 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:40.488 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:32:40.489 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:32:45.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:45.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:45.492 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:32:45.533 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:32:45.533 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:32:45.667 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:32:45.667 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:32:45.667 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:32:45.667 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:32:45.668 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:32:45.668 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:32:45.668 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:45.668 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:45.669 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:32:45.669 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:45.669 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:32:45.669 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:45.669 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:45.670 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:32:45.670 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:45.670 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:32:45.670 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:45.671 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:32:45.671 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:45.671 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:45.671 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:45.672 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:45.672 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:32:45.672 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:45.672 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:45.672 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.240117_333.1", 2021-09-17T10:32:45.673 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:45.673 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:45.673 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:45.673 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.240117Z", 2021-09-17T10:32:45.674 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:45.674 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:45.674 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:45.674 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:45.674 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:32:45.675 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:45.675 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:45.675 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:32:45.675 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:45.676 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:45.676 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:45.676 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:32:45.676 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:45.676 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:45.677 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:45.677 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:45.677 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:32:45.677 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:45.678 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:45.678 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:32:45.678 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:45.678 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:45.678 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:45.679 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:32:45.679 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:45.679 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:45.679 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:45.680 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:32:45.680 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:45.680 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:32:45.680 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:32:45.680 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:32:45.681 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:45.681 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:32:45.682 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.240117_333.1 2021-09-17T10:32:50.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:50.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:50.685 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:32:50.780 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:32:50.780 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:32:51.019 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:32:51.020 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:32:51.020 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:32:51.020 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:32:51.020 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:32:51.021 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:32:51.021 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:51.021 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:51.021 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:32:51.021 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:51.022 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:32:51.022 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:51.022 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:51.022 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:32:51.023 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:51.023 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:32:51.023 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:51.023 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:32:51.024 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:51.024 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:51.024 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:51.024 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:51.024 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:32:51.025 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:51.025 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:51.025 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:32:51.025 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:51.026 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:51.026 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:51.026 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:32:51.026 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:51.026 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:51.027 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:51.027 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:51.027 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:32:51.027 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:51.028 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:51.028 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:32:51.028 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:51.028 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:51.028 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:51.029 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:32:51.029 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:51.029 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:51.029 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:51.029 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:51.030 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:32:51.030 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:51.030 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:51.030 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:32:51.031 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:51.031 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:51.031 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:51.031 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:32:51.032 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:51.032 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:51.032 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:51.032 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:32:51.032 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:51.033 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:32:51.033 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:32:51.033 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:32:51.033 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:51.034 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:32:51.035 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:32:56.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:56.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:32:56.038 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:32:56.080 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:32:56.080 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:32:56.202 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:32:56.203 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:32:56.203 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:32:56.203 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:32:56.203 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:32:56.204 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:32:56.204 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:56.204 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:56.205 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:32:56.205 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:56.205 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:32:56.205 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:56.206 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:56.206 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:32:56.206 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:56.206 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:32:56.206 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:56.207 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:32:56.207 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:56.207 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:56.207 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:56.208 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:56.208 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:32:56.208 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:56.208 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:56.209 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.001301_323.1", 2021-09-17T10:32:56.209 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:56.209 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:56.210 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:56.210 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.001301Z", 2021-09-17T10:32:56.210 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:56.210 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:56.210 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:56.211 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:56.211 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:32:56.211 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:56.211 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:56.212 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:32:56.212 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:56.212 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:56.212 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:56.213 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:32:56.213 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:56.213 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:56.213 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:56.213 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:32:56.214 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:32:56.214 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:32:56.214 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:32:56.214 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:32:56.215 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:32:56.215 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:32:56.215 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:32:56.215 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:32:56.215 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:32:56.216 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:56.216 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:56.216 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:32:56.216 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:32:56.217 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:32:56.217 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:32:56.217 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:32:56.217 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:32:56.217 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:32:56.219 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.001301_323.1 2021-09-17T10:33:01.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:01.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:01.221 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:33:01.265 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:33:01.266 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:33:01.672 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:33:01.672 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:33:01.673 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:33:01.673 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:33:01.673 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:33:01.673 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:33:01.674 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:01.674 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:01.674 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:33:01.674 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:01.675 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:33:01.675 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:01.675 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:01.675 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:33:01.676 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:01.676 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:33:01.676 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:01.676 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:33:01.676 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:01.677 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:01.677 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:01.677 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:01.677 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:33:01.678 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:01.678 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:01.678 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.001301_323.1", 2021-09-17T10:33:01.678 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:01.679 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:01.679 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:01.679 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.001301Z", 2021-09-17T10:33:01.679 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:01.680 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:01.680 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:01.680 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:01.680 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:33:01.681 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:01.681 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:01.681 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:01.681 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:01.681 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:01.682 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:01.682 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:01.682 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:01.682 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:01.684 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:01.684 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:01.684 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:33:01.685 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:01.685 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:01.686 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:01.686 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:01.686 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:01.686 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:01.687 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:01.687 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:01.688 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:01.688 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:01.688 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:33:01.688 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:01.689 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:33:01.689 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:33:01.689 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:33:01.689 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:01.689 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:33:01.691 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.001301_323.1 2021-09-17T10:33:06.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:06.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:06.693 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:33:06.755 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:33:06.756 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:33:06.930 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:33:06.930 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:33:06.930 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:33:06.931 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:33:06.931 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:33:06.931 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:33:06.931 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:06.931 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:06.932 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:33:06.932 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:06.932 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:33:06.932 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:06.932 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:06.933 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:33:06.933 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:06.933 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:33:06.933 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:06.933 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:33:06.934 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:06.934 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:06.934 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:06.934 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:06.934 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:33:06.935 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:06.935 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:06.935 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:33:06.935 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:06.935 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:06.936 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:06.936 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:33:06.936 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:06.936 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:06.936 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:06.937 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:06.937 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:33:06.937 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:06.937 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:06.937 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:06.938 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:06.938 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:06.938 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:06.938 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:06.939 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:06.939 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:06.939 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:06.939 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:06.939 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:33:06.940 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:06.940 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:06.940 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:06.940 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:06.941 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:06.941 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:06.941 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:06.941 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:06.941 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:06.942 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:06.942 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:33:06.942 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:06.942 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:33:06.942 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:33:06.943 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:33:06.943 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:06.943 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:33:06.944 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:33:11.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:11.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:11.946 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:33:11.991 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:33:11.992 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:33:12.113 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:33:12.114 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:33:12.114 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:33:12.114 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:33:12.115 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:33:12.115 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:33:12.115 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:12.115 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:12.115 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:33:12.116 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:12.116 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:33:12.116 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:12.116 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:12.117 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:33:12.117 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:12.117 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:33:12.117 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:12.117 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:33:12.118 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:12.118 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:12.118 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:12.118 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:12.119 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:33:12.119 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:12.119 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:12.119 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:33:12.120 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:12.120 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:12.120 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:12.120 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:33:12.120 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:12.121 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:12.121 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:12.121 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:12.121 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:33:12.122 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:12.122 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:12.122 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:12.122 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:12.123 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:12.123 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:12.123 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:12.123 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:12.124 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:12.124 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:12.124 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:12.124 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:33:12.125 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:12.125 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:12.125 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:12.125 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:12.125 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:12.126 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:12.126 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:12.126 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:12.127 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:12.127 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:12.127 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:33:12.127 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:12.127 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:33:12.128 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:33:12.128 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:33:12.128 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:12.128 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:33:12.130 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:33:17.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:17.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:17.132 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:33:17.176 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:33:17.177 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:33:17.604 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:33:17.605 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:33:17.605 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:33:17.605 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:33:17.606 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:33:17.606 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:33:17.606 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:17.606 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:17.607 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:33:17.607 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:17.607 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:33:17.607 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:17.607 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:17.608 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:33:17.608 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:17.608 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:33:17.608 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:17.609 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:33:17.609 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:17.609 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:17.609 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:17.609 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:17.610 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:33:17.610 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:17.610 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:17.610 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.099168_329.1", 2021-09-17T10:33:17.611 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:17.611 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:17.611 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:17.611 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.099168Z", 2021-09-17T10:33:17.612 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:17.612 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:17.612 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:17.612 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:17.612 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:33:17.613 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:17.613 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:17.613 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:17.613 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:17.614 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:17.614 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:17.614 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:17.614 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:17.614 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:17.615 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:17.615 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:17.615 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:33:17.615 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:17.616 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:17.616 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:17.616 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:17.616 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:17.616 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:17.617 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:17.617 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:17.617 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:17.617 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:17.618 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:33:17.618 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:17.618 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:33:17.619 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:33:17.619 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:33:17.619 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:17.619 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:33:17.620 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.099168_329.1 2021-09-17T10:33:22.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:22.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:22.623 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:33:22.718 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:33:22.719 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:33:23.058 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:33:23.059 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:33:23.059 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:33:23.059 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:33:23.059 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:33:23.059 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:33:23.060 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:23.060 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:23.060 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:33:23.060 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:23.060 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:33:23.061 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:23.061 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:23.061 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:33:23.061 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:23.061 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:33:23.062 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:23.062 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:33:23.062 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:23.062 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:23.062 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:23.063 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:23.063 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:33:23.063 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:23.063 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:23.063 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:33:23.064 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:23.064 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:23.064 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:23.064 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:33:23.064 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:23.065 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:23.065 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:23.065 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:23.065 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:33:23.066 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:23.066 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:23.066 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:23.066 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:23.066 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:23.067 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:23.067 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:23.067 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:23.067 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:23.067 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:23.068 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:23.068 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:33:23.068 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:23.068 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:23.068 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:23.069 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:23.069 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:23.069 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:23.069 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:23.070 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:23.070 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:23.070 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:23.070 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:33:23.070 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:23.071 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:33:23.071 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:33:23.071 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:33:23.071 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:23.071 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:33:23.072 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:33:28.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:28.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:28.075 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:33:28.106 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:33:28.106 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:33:28.214 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:33:28.214 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:33:28.215 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:33:28.215 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:33:28.215 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:33:28.215 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:33:28.216 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:28.216 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:28.216 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:33:28.216 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:28.217 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:33:28.217 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:28.217 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:28.217 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:33:28.218 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:28.218 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:33:28.218 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:28.218 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:33:28.218 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:28.219 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:28.219 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:28.219 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:28.219 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:33:28.220 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:28.220 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:28.220 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.105095_330.1", 2021-09-17T10:33:28.220 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:28.221 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:28.221 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:28.221 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.105095Z", 2021-09-17T10:33:28.221 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:28.222 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:28.222 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:28.222 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:28.222 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:33:28.223 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:28.223 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:28.223 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:28.223 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:28.224 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:28.224 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:28.224 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:28.224 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:28.225 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:28.225 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:28.225 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:28.225 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:33:28.225 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:28.226 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:28.226 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:28.226 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:28.226 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:28.227 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:28.227 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:28.227 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:28.227 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:28.228 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:28.228 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:33:28.228 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:28.228 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:33:28.229 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:33:28.229 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:33:28.229 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:28.229 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:33:28.231 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.105095_330.1 2021-09-17T10:33:33.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:33.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:33.234 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:33:33.332 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:33:33.332 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:33:33.816 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:33:33.816 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:33:33.816 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:33:33.817 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:33:33.817 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:33:33.817 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:33:33.817 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:33.818 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:33.818 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:33:33.818 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:33.818 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:33:33.818 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:33.819 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:33.819 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:33:33.819 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:33.819 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:33:33.819 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:33.820 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:33:33.820 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:33.820 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:33.820 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:33.821 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:33.821 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:33:33.821 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:33.821 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:33.821 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.240117_333.1", 2021-09-17T10:33:33.822 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:33.822 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:33.822 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:33.822 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.240117Z", 2021-09-17T10:33:33.822 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:33.823 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:33.823 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:33.823 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:33.823 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:33:33.824 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:33.824 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:33.824 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:33.824 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:33.824 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:33.825 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:33.825 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:33.825 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:33.825 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:33.825 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:33.826 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:33.826 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:33:33.826 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:33.826 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:33.827 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:33.827 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:33.827 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:33.827 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:33.827 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:33.828 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:33.828 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:33.828 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:33.828 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:33:33.828 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:33.828 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:33:33.829 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:33:33.829 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:33:33.829 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:33.829 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:33:33.830 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.240117_333.1 2021-09-17T10:33:38.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:38.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:38.834 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:33:38.874 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:33:38.875 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:33:38.966 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:33:38.966 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:33:38.967 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:33:38.967 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:33:38.967 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:33:38.967 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:33:38.967 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:38.968 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:38.968 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:33:38.968 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:38.968 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:33:38.968 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:38.969 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:38.969 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:33:38.969 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:38.969 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:33:38.970 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:38.970 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:33:38.970 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:38.970 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:38.970 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:38.971 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:38.971 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:33:38.971 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:38.971 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:38.971 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:33:38.972 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:38.972 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:38.972 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:38.972 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:33:38.972 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:38.973 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:38.973 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:38.973 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:38.973 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:33:38.973 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:38.974 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:38.974 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:38.974 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:38.974 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:38.975 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:38.975 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:38.975 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:38.975 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:38.975 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:38.976 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:38.976 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:33:38.976 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:38.976 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:38.976 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:38.977 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:38.977 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:38.977 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:38.977 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:38.977 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:38.978 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:38.978 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:38.978 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:33:38.978 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:38.978 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:33:38.979 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:33:38.979 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:33:38.979 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:38.979 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:33:38.980 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:33:43.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:43.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:43.982 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:33:44.025 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:33:44.025 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:33:44.146 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:33:44.147 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:33:44.147 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:33:44.147 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:33:44.147 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:33:44.147 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:33:44.148 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:44.148 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:44.148 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:33:44.148 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:44.149 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:33:44.149 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:44.149 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:44.149 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:33:44.149 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:44.150 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:33:44.150 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:44.150 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:33:44.150 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:44.150 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:44.151 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:44.151 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:44.151 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:33:44.151 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:44.151 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:44.152 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.256150_335.1", 2021-09-17T10:33:44.152 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:44.152 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:44.152 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:44.152 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.256150Z", 2021-09-17T10:33:44.153 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:44.153 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:44.153 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:44.153 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:44.153 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:33:44.154 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:44.154 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:44.154 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:44.154 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:44.155 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:44.155 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:44.155 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:44.155 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:44.155 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:44.156 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:44.156 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:44.156 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:33:44.156 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:44.156 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:44.157 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:44.157 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:44.157 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:44.157 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:44.157 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:44.158 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:44.158 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:44.158 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:44.158 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:33:44.158 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:44.159 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:33:44.159 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:33:44.159 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:33:44.159 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:44.159 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:33:44.160 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.256150_335.1 2021-09-17T10:33:49.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:49.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:49.163 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:33:49.222 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:33:49.222 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:33:49.604 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:33:49.605 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:33:49.605 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:33:49.605 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:33:49.605 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:33:49.606 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:33:49.606 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:49.606 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:49.606 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:33:49.606 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:49.607 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:33:49.607 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:49.607 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:49.607 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:33:49.608 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:49.608 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:33:49.608 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:49.608 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:33:49.609 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:49.609 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:49.609 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:49.609 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:49.609 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:33:49.610 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:49.610 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:49.610 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.959198_322.1", 2021-09-17T10:33:49.610 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:49.611 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:49.611 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:49.611 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.959198Z", 2021-09-17T10:33:49.611 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:49.611 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:49.612 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:49.612 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:49.612 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:33:49.612 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:49.613 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:49.613 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:49.613 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:49.613 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:49.613 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:49.614 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:49.614 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:49.614 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:49.614 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:49.615 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:49.615 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:33:49.615 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:49.615 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:49.616 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:49.616 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:49.616 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:49.616 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:49.616 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:49.617 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:49.617 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:49.617 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:49.617 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:33:49.618 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:49.618 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:33:49.618 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:33:49.618 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:33:49.618 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:49.619 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:33:49.620 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.959198_322.1 2021-09-17T10:33:54.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:54.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:54.623 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:33:54.665 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:33:54.665 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:33:54.789 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:33:54.790 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:33:54.790 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:33:54.790 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:33:54.791 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:33:54.791 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:33:54.791 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:54.791 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:54.792 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:33:54.792 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:54.792 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:33:54.792 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:54.792 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:54.793 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:33:54.793 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:54.793 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:33:54.793 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:54.794 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:33:54.794 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:54.794 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:54.794 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:54.795 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:54.795 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:33:54.795 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:54.795 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:54.795 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.105095_330.1", 2021-09-17T10:33:54.796 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:54.796 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:54.796 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:54.796 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.105095Z", 2021-09-17T10:33:54.797 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:54.797 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:54.797 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:54.797 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:54.797 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:33:54.798 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:54.798 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:54.798 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:54.798 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:54.799 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:54.799 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:54.799 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:54.799 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:54.800 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:54.800 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:54.800 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:33:54.800 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:33:54.800 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:33:54.801 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:33:54.801 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:33:54.801 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:33:54.801 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:33:54.801 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:33:54.802 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:33:54.802 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:33:54.802 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:54.802 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:54.803 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:33:54.803 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:33:54.803 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:33:54.803 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:33:54.803 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:33:54.804 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:33:54.804 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:33:54.805 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.105095_330.1 2021-09-17T10:33:59.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:59.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:33:59.808 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:33:59.850 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:33:59.850 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:34:00.044 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:34:00.044 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:34:00.044 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:34:00.045 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:34:00.045 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:34:00.045 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:34:00.045 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:00.046 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:00.046 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:34:00.046 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:00.046 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:34:00.047 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:00.047 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:00.047 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:34:00.047 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:00.047 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:34:00.048 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:00.048 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:34:00.048 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:00.048 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:00.049 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:00.049 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:00.049 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:34:00.049 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:00.049 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:00.050 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.344112_340.1", 2021-09-17T10:34:00.050 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:00.050 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:00.050 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:00.050 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.344112Z", 2021-09-17T10:34:00.051 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:00.051 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:00.051 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:00.051 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:00.052 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:34:00.052 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:00.052 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:00.052 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:00.053 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:00.053 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:00.053 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:00.053 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:00.053 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:00.054 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:00.054 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:00.054 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:00.054 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:34:00.055 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:00.055 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:00.055 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:00.055 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:00.055 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:00.056 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:00.056 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:00.056 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:00.056 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:00.057 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:00.057 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:34:00.057 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:00.057 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:34:00.057 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:34:00.058 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:34:00.058 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:00.058 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:34:00.059 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.344112_340.1 2021-09-17T10:34:05.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:05.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:05.062 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:34:05.129 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:34:05.129 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:34:05.368 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:34:05.369 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:34:05.369 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:34:05.369 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:34:05.370 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:34:05.370 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:34:05.370 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:05.370 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:05.370 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:34:05.371 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:05.371 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:34:05.371 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:05.371 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:05.372 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:34:05.372 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:05.372 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:34:05.372 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:05.373 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:34:05.373 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:05.373 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:05.373 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:05.373 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:05.374 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:34:05.374 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:05.374 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:05.374 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.007158_324.1", 2021-09-17T10:34:05.375 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:05.375 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:05.375 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:05.375 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.007158Z", 2021-09-17T10:34:05.376 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:05.376 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:05.376 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:05.376 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:05.376 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:34:05.377 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:05.377 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:05.377 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:05.377 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:05.378 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:05.378 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:05.378 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:05.378 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:05.378 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:05.379 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:05.379 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:05.379 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:34:05.379 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:05.380 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:05.380 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:05.380 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:05.380 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:05.381 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:05.381 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:05.381 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:05.381 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:05.381 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:05.382 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:34:05.382 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:05.382 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:34:05.382 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:34:05.383 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:34:05.383 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:05.383 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:34:05.384 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.007158_324.1 2021-09-17T10:34:10.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:10.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:10.387 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:34:10.429 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:34:10.430 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:34:10.550 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:34:10.550 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:34:10.551 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:34:10.551 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:34:10.551 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:34:10.551 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:34:10.551 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:10.552 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:10.552 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:34:10.552 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:10.553 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:34:10.553 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:10.554 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:10.554 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:34:10.554 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:10.555 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:34:10.555 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:10.555 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:34:10.556 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:10.556 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:10.556 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:10.556 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:10.557 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:34:10.557 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:10.557 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:10.557 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:34:10.558 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:10.558 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:10.558 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:10.558 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:34:10.558 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:10.559 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:10.559 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:10.559 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:10.559 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:34:10.559 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:10.560 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:10.560 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:10.560 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:10.560 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:10.561 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:10.561 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:10.561 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:10.561 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:10.561 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:10.562 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:10.562 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:34:10.562 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:10.562 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:10.562 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:10.563 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:10.563 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:10.563 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:10.563 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:10.564 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:10.564 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:10.564 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:10.564 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:34:10.564 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:10.565 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:34:10.565 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:34:10.565 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:34:10.565 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:10.565 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:34:10.566 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:34:15.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:15.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:15.569 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:34:15.671 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:34:15.671 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:34:16.085 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:34:16.086 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:34:16.086 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:34:16.086 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:34:16.087 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:34:16.087 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:34:16.087 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:16.087 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:16.088 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:34:16.088 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:16.088 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:34:16.088 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:16.089 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:16.089 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:34:16.089 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:16.089 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:34:16.090 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:16.090 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:34:16.090 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:16.090 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:16.091 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:16.091 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:16.091 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:34:16.091 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:16.091 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:16.092 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.105095_330.1", 2021-09-17T10:34:16.092 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:16.092 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:16.092 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:16.093 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.105095Z", 2021-09-17T10:34:16.093 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:16.093 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:16.093 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:16.094 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:16.094 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:34:16.094 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:16.094 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:16.095 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:16.095 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:16.095 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:16.096 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:16.096 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:16.096 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:16.096 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:16.097 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:16.097 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:16.097 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:34:16.097 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:16.098 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:16.098 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:16.098 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:16.098 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:16.099 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:16.099 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:16.099 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:16.099 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:16.100 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:16.100 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:34:16.100 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:16.100 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:34:16.101 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:34:16.101 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:34:16.101 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:16.101 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:34:16.102 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.105095_330.1 2021-09-17T10:34:21.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:21.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:21.105 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:34:21.157 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:34:21.157 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:34:21.251 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:34:21.252 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:34:21.252 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:34:21.252 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:34:21.252 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:34:21.252 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:34:21.253 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:21.253 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:21.253 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:34:21.253 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:21.254 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:34:21.254 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:21.254 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:21.254 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:34:21.254 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:21.255 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:34:21.255 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:21.255 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:34:21.255 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:21.255 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:21.256 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:21.256 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:21.256 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:34:21.256 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:21.257 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:21.257 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:34:21.257 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:21.257 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:21.257 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:21.258 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:34:21.258 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:21.258 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:21.258 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:21.258 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:21.259 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:34:21.259 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:21.259 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:21.259 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:21.259 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:21.260 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:21.260 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:21.260 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:21.260 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:21.260 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:21.261 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:21.261 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:21.261 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:34:21.261 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:21.261 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:21.262 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:21.262 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:21.262 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:21.262 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:21.262 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:21.263 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:21.263 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:21.263 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:21.263 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:34:21.263 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:21.264 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:34:21.264 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:34:21.264 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:34:21.264 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:21.265 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:34:21.266 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:34:26.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:26.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:26.268 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:34:26.312 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:34:26.312 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:34:26.436 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:34:26.436 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:34:26.437 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:34:26.437 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:34:26.437 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:34:26.438 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:34:26.438 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:26.438 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:26.438 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:34:26.438 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:26.439 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:34:26.439 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:26.439 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:26.439 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:34:26.440 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:26.440 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:34:26.440 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:26.440 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:34:26.440 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:26.441 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:26.441 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:26.441 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:26.441 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:34:26.442 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:26.442 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:26.442 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:34:26.442 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:26.443 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:26.443 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:26.443 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:34:26.443 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:26.443 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:26.444 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:26.444 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:26.444 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:34:26.444 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:26.444 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:26.445 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:26.445 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:26.445 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:26.445 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:26.446 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:26.446 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:26.446 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:26.446 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:26.446 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:26.447 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:34:26.447 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:26.447 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:26.447 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:26.448 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:26.448 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:26.448 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:26.448 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:26.448 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:26.449 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:26.449 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:26.449 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:34:26.449 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:26.449 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:34:26.450 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:34:26.450 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:34:26.450 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:26.450 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:34:26.451 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:34:31.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:31.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:31.454 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:34:31.617 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:34:31.617 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:34:32.105 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:34:32.105 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:34:32.105 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:34:32.105 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:34:32.106 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:34:32.106 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:34:32.106 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:32.106 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:32.107 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:34:32.107 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:32.107 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:34:32.107 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:32.107 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:32.108 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:34:32.108 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:32.108 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:34:32.108 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:32.109 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:34:32.109 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:32.109 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:32.109 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:32.109 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:32.110 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:34:32.110 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:32.110 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:32.110 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:34:32.111 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:32.111 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:32.111 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:32.111 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:34:32.112 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:32.112 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:32.112 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:32.112 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:32.112 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:34:32.113 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:32.113 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:32.113 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:32.113 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:32.113 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:32.114 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:32.114 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:32.114 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:32.114 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:32.114 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:32.115 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:32.115 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:34:32.115 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:32.115 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:32.116 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:32.116 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:32.116 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:32.116 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:32.116 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:32.117 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:32.117 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:32.117 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:32.117 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:34:32.117 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:32.118 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:34:32.118 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:34:32.118 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:34:32.118 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:32.118 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:34:32.119 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:34:37.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:37.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:37.121 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:34:37.161 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:34:37.161 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:34:37.272 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:34:37.272 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:34:37.272 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:34:37.272 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:34:37.273 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:34:37.273 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:34:37.273 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:37.273 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:37.274 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:34:37.274 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:37.274 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:34:37.274 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:37.274 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:37.275 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:34:37.275 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:37.275 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:34:37.275 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:37.275 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:34:37.276 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:37.276 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:37.276 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:37.276 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:37.277 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:34:37.277 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:37.277 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:37.277 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:34:37.277 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:37.278 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:37.278 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:37.278 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:34:37.278 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:37.279 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:37.279 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:37.279 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:37.279 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:34:37.279 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:37.280 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:37.280 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:37.280 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:37.280 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:37.281 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:37.281 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:37.281 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:37.281 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:37.281 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:37.282 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:37.282 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:34:37.282 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:37.282 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:37.282 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:37.283 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:37.283 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:37.283 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:37.283 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:37.283 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:37.284 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:37.284 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:37.284 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:34:37.284 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:37.285 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:34:37.285 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:34:37.285 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:34:37.285 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:37.285 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:34:37.286 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.952206_320.1 2021-09-17T10:34:42.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:42.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:42.289 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:34:42.331 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:34:42.331 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:34:42.455 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:34:42.455 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:34:42.455 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:34:42.455 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:34:42.456 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:34:42.456 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:34:42.456 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:42.456 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:42.457 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:34:42.457 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:42.457 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:34:42.457 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:42.457 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:42.458 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:34:42.458 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:42.458 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:34:42.458 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:42.459 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:34:42.459 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:42.459 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:42.459 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:42.459 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:42.460 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:34:42.460 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:42.460 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:42.460 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.249312_334.1", 2021-09-17T10:34:42.461 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:42.461 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:42.461 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:42.461 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.249312Z", 2021-09-17T10:34:42.461 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:42.462 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:42.462 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:42.462 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:42.462 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:34:42.463 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:42.463 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:42.463 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:42.463 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:42.463 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:42.464 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:42.464 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:42.464 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:42.464 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:42.464 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:42.465 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:42.465 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:34:42.465 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:42.465 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:42.466 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:42.466 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:42.466 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:42.466 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:42.466 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:42.467 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:42.467 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:42.467 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:42.467 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:34:42.468 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:42.468 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:34:42.468 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:34:42.468 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:34:42.468 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:42.469 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:34:42.470 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.249312_334.1 2021-09-17T10:34:47.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:47.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:47.472 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:34:47.561 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:34:47.561 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:34:47.944 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:34:47.945 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:34:47.945 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:34:47.945 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:34:47.945 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:34:47.946 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:34:47.946 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:47.946 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:47.946 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:34:47.947 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:47.947 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:34:47.947 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:47.947 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:47.948 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:34:47.948 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:47.948 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:34:47.948 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:47.949 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:34:47.949 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:47.949 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:47.949 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:47.949 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:47.950 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:34:47.950 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:47.950 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:47.950 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:34:47.951 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:47.951 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:47.951 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:47.951 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:34:47.952 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:47.952 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:47.952 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:47.952 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:47.952 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:34:47.953 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:47.953 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:47.953 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:47.953 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:47.954 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:47.954 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:47.954 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:47.954 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:47.955 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:47.955 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:47.955 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:47.955 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:34:47.955 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:47.956 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:47.956 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:47.956 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:47.956 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:47.957 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:47.957 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:47.957 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:47.957 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:47.958 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:47.958 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:34:47.958 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:47.958 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:34:47.959 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:34:47.959 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:34:47.959 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:47.959 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:34:47.960 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:34:52.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:52.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:52.964 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:34:53.004 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:34:53.004 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:34:53.127 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:34:53.128 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:34:53.128 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:34:53.128 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:34:53.128 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:34:53.128 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:34:53.129 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:53.129 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:53.129 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:34:53.129 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:53.130 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:34:53.130 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:53.130 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:53.130 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:34:53.130 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:53.131 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:34:53.131 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:53.131 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:34:53.131 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:53.132 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:53.132 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:53.132 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:53.132 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:34:53.132 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:53.133 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:53.133 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.099168_329.1", 2021-09-17T10:34:53.133 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:53.133 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:53.133 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:53.134 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.099168Z", 2021-09-17T10:34:53.134 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:53.134 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:53.134 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:53.135 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:53.135 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:34:53.135 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:53.135 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:53.135 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:53.136 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:53.136 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:53.136 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:53.136 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:53.137 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:53.137 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:53.137 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:53.137 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:53.138 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:34:53.138 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:53.138 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:53.138 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:53.138 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:53.139 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:53.139 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:53.139 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:53.139 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:53.140 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:53.140 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:53.140 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:34:53.140 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:53.141 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:34:53.141 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:34:53.141 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:34:53.141 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:53.141 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:34:53.143 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.099168_329.1 2021-09-17T10:34:58.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:58.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:34:58.145 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:34:58.188 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:34:58.189 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:34:58.316 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:34:58.317 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:34:58.317 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:34:58.317 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:34:58.317 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:34:58.318 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:34:58.318 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:58.318 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:58.318 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:34:58.318 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:58.319 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:34:58.319 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:58.319 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:58.319 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:34:58.320 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:58.320 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:34:58.320 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:58.320 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:34:58.321 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:58.321 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:58.321 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:58.321 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:58.321 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:34:58.322 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:58.322 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:58.322 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.007158_324.1", 2021-09-17T10:34:58.322 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:58.323 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:58.323 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:58.323 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.007158Z", 2021-09-17T10:34:58.323 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:58.324 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:58.324 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:58.324 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:58.324 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:34:58.324 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:58.325 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:58.325 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:58.325 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:58.325 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:58.325 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:58.326 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:58.326 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:58.326 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:58.326 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:58.326 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:34:58.327 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:34:58.327 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:34:58.327 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:34:58.327 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:34:58.328 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:34:58.328 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:34:58.328 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:34:58.328 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:34:58.328 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:34:58.329 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:58.329 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:58.329 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:34:58.329 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:34:58.329 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:34:58.330 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:34:58.330 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:34:58.330 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:34:58.330 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:34:58.331 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.007158_324.1 2021-09-17T10:35:03.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:03.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:03.335 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:35:03.397 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:35:03.398 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:35:03.708 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:35:03.708 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:35:03.709 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:35:03.709 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:35:03.709 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:35:03.709 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:35:03.710 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:03.710 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:03.710 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:35:03.711 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:03.711 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:35:03.711 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:03.711 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:03.712 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:35:03.712 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:03.712 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:35:03.713 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:03.713 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:35:03.713 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:03.713 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:35:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:03.715 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:03.715 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:35:03.715 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:03.715 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:03.716 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:03.716 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:35:03.716 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:03.716 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:03.716 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:03.717 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:03.717 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:35:03.717 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:03.717 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:03.718 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:03.718 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:03.718 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:03.718 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:03.718 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:03.719 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:03.719 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:03.719 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:03.719 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:03.720 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:35:03.720 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:03.720 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:03.720 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:03.721 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:03.721 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:03.721 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:03.721 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:03.721 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:03.722 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:03.722 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:03.722 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:35:03.722 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:03.723 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:35:03.723 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:35:03.723 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:35:03.723 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:03.723 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:35:03.725 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:35:08.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:08.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:08.728 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:35:08.769 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:35:08.770 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:35:08.887 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:35:08.887 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:35:08.888 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:35:08.888 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:35:08.888 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:35:08.888 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:35:08.889 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:08.889 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:08.889 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:35:08.889 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:08.890 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:35:08.890 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:08.890 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:08.890 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:35:08.891 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:08.891 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:35:08.891 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:08.891 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:35:08.892 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:08.892 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:08.892 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:08.892 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:08.893 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:35:08.893 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:08.893 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:08.893 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.334371_338.1", 2021-09-17T10:35:08.893 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:08.894 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:08.894 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:08.894 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.334371Z", 2021-09-17T10:35:08.894 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:08.895 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:08.895 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:08.895 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:08.895 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:35:08.896 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:08.896 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:08.896 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:08.896 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:08.896 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:08.897 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:08.897 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:08.897 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:08.897 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:08.898 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:08.898 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:08.898 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:35:08.898 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:08.899 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:08.899 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:08.899 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:08.899 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:08.900 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:08.900 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:08.900 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:08.900 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:08.900 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:08.901 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:35:08.901 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:08.901 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:35:08.901 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:35:08.902 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:35:08.902 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:08.902 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:35:08.903 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.334371_338.1 2021-09-17T10:35:13.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:13.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:13.907 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:35:13.948 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:35:13.948 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:35:14.079 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:35:14.079 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:35:14.080 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:35:14.080 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:35:14.080 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:35:14.080 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:35:14.080 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:14.081 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:14.081 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:35:14.081 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:14.081 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:35:14.082 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:14.082 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:14.082 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:35:14.082 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:14.083 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:35:14.083 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:14.083 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:35:14.083 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:14.084 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:14.084 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:14.084 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:14.084 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:35:14.084 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:14.085 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:14.085 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:35:14.085 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:14.085 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:14.086 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:14.086 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:35:14.086 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:14.086 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:14.087 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:14.087 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:14.087 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:35:14.087 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:14.088 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:14.088 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:14.088 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:14.088 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:14.088 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:14.089 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:14.089 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:14.089 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:14.089 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:14.090 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:14.090 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:35:14.090 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:14.090 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:14.091 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:14.091 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:14.091 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:14.091 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:14.092 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:14.092 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:14.092 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:14.092 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:14.092 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:35:14.093 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:14.093 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:35:14.093 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:35:14.093 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:35:14.094 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:14.094 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:35:14.095 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:35:19.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:19.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:19.097 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:35:19.183 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:35:19.183 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:35:19.533 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:35:19.533 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:35:19.534 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:35:19.534 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:35:19.534 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:35:19.535 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:35:19.535 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:19.535 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:19.535 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:35:19.536 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:19.536 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:35:19.536 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:19.536 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:19.536 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:35:19.537 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:19.537 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:35:19.537 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:19.537 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:35:19.538 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:19.538 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:19.538 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:19.538 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:19.539 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:35:19.539 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:19.539 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:19.539 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:35:19.539 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:19.540 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:19.540 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:19.540 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:35:19.540 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:19.541 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:19.541 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:19.541 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:19.541 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:35:19.541 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:19.542 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:19.542 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:19.542 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:19.542 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:19.543 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:19.543 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:19.543 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:19.543 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:19.543 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:19.544 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:19.544 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:35:19.544 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:19.544 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:19.545 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:19.545 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:19.545 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:19.545 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:19.545 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:19.546 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:19.546 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:19.546 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:19.546 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:35:19.547 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:19.547 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:35:19.547 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:35:19.547 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:35:19.547 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:19.548 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:35:19.548 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:35:24.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:24.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:24.551 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:35:24.593 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:35:24.593 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:35:24.722 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:35:24.722 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:35:24.722 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:35:24.723 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:35:24.723 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:35:24.723 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:35:24.723 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:24.724 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:24.724 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:35:24.724 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:24.724 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:35:24.725 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:24.725 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:24.725 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:35:24.725 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:24.726 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:35:24.726 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:24.726 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:35:24.727 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:24.727 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:24.727 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:24.727 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:24.727 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:35:24.728 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:24.728 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:24.728 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:35:24.728 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:24.729 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:24.729 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:24.729 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:35:24.729 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:24.730 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:24.730 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:24.730 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:24.730 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:35:24.731 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:24.731 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:24.731 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:24.731 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:24.732 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:24.732 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:24.732 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:24.732 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:24.732 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:24.733 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:24.733 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:24.733 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:35:24.733 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:24.734 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:24.734 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:24.734 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:24.734 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:24.735 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:24.735 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:24.735 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:24.735 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:24.735 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:24.736 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:35:24.736 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:24.736 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:35:24.736 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:35:24.737 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:35:24.737 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:24.737 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:35:24.738 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:35:29.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:29.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:29.741 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:35:29.787 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:35:29.787 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:35:30.203 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:35:30.203 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:35:30.204 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:35:30.204 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:35:30.204 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:35:30.204 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:35:30.205 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:30.205 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:30.205 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:35:30.205 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:30.205 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:35:30.206 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:30.206 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:30.206 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:35:30.206 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:30.207 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:35:30.207 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:30.207 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:35:30.207 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:30.207 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:30.208 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:30.208 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:30.208 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:35:30.208 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:30.208 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:30.209 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.014731_326.1", 2021-09-17T10:35:30.209 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:30.209 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:30.209 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:30.210 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.014731Z", 2021-09-17T10:35:30.210 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:30.210 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:30.210 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:30.211 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:30.211 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:35:30.211 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:30.211 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:30.211 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:30.212 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:30.212 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:30.212 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:30.212 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:30.213 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:30.213 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:30.213 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:30.213 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:30.213 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:35:30.214 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:30.214 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:30.214 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:30.214 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:30.215 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:30.215 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:30.215 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:30.215 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:30.215 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:30.216 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:30.216 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:35:30.216 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:30.216 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:35:30.216 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:35:30.217 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:35:30.217 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:30.217 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:35:30.217 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.014731_326.1 2021-09-17T10:35:35.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:35.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:35.220 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:35:35.262 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:35:35.262 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:35:35.352 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:35:35.352 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:35:35.353 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:35:35.353 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:35:35.353 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:35:35.354 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:35:35.354 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:35.354 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:35.354 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:35:35.355 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:35.355 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:35:35.355 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:35.355 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:35.356 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:35:35.356 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:35.356 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:35:35.356 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:35.357 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:35:35.357 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:35.357 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:35.357 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:35.358 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:35.358 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:35:35.358 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:35.358 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:35.359 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.249312_334.1", 2021-09-17T10:35:35.359 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:35.359 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:35.359 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:35.360 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.249312Z", 2021-09-17T10:35:35.360 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:35.360 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:35.360 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:35.361 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:35.361 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:35:35.361 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:35.361 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:35.362 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:35.362 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:35.362 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:35.362 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:35.363 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:35.363 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:35.363 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:35.363 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:35.364 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:35.364 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:35:35.364 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:35.364 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:35.365 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:35.365 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:35.365 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:35.365 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:35.366 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:35.366 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:35.366 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:35.366 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:35.367 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:35:35.367 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:35.367 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:35:35.367 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:35:35.368 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:35:35.368 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:35.368 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:35:35.369 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.249312_334.1 2021-09-17T10:35:40.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:40.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:40.372 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:35:40.418 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:35:40.418 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:35:40.537 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:35:40.538 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:35:40.538 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:35:40.538 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:35:40.538 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:35:40.539 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:35:40.539 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:40.539 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:40.539 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:35:40.540 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:40.540 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:35:40.540 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:40.540 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:40.540 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:35:40.541 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:40.541 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:35:40.541 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:40.541 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:35:40.541 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:40.542 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:40.542 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:40.542 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:40.542 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:35:40.542 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:40.543 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:40.543 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:35:40.543 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:40.543 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:40.543 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:40.544 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:35:40.544 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:40.544 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:40.544 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:40.545 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:40.545 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:35:40.545 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:40.545 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:40.545 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:40.545 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:40.546 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:40.546 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:40.546 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:40.546 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:40.547 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:40.547 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:40.547 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:40.547 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:35:40.548 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:40.548 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:40.548 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:40.548 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:40.548 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:40.549 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:40.549 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:40.549 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:40.549 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:40.549 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:40.550 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:35:40.550 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:40.550 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:35:40.550 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:35:40.550 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:35:40.551 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:40.551 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:35:40.552 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:35:45.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:45.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:45.555 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:35:45.676 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:35:45.676 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:35:46.020 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:35:46.020 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:35:46.020 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:35:46.021 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:35:46.021 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:35:46.021 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:35:46.021 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:46.021 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:46.022 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:35:46.022 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:46.022 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:35:46.022 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:46.023 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:46.023 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:35:46.023 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:46.023 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:35:46.023 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:46.024 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:35:46.024 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:46.024 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:46.024 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:46.024 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:46.025 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:35:46.025 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:46.025 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:46.025 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:35:46.025 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:46.026 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:46.026 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:46.026 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:35:46.026 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:46.027 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:46.027 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:46.027 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:46.027 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:35:46.027 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:46.028 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:46.028 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:46.028 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:46.028 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:46.028 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:46.029 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:46.029 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:46.029 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:46.029 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:46.029 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:46.030 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:35:46.030 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:46.030 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:46.030 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:46.031 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:46.031 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:46.031 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:46.031 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:46.031 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:46.032 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:46.032 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:46.032 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:35:46.032 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:46.032 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:35:46.033 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:35:46.033 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:35:46.033 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:46.033 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:35:46.034 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:35:51.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:51.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:51.036 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:35:51.082 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:35:51.082 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:35:51.165 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:35:51.165 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:35:51.165 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:35:51.166 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:35:51.166 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:35:51.166 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:35:51.166 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:51.166 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:51.167 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:35:51.167 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:51.167 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:35:51.167 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:51.168 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:51.168 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:35:51.168 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:51.168 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:35:51.168 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:51.169 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:35:51.169 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:51.169 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:51.169 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:51.169 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:51.170 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:35:51.170 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:51.170 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:51.170 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.007158_324.1", 2021-09-17T10:35:51.170 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:51.171 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:51.171 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:51.171 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.007158Z", 2021-09-17T10:35:51.171 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:51.171 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:51.172 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:51.172 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:51.172 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:35:51.172 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:51.172 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:51.173 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:51.173 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:51.173 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:51.173 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:51.173 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:51.174 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:51.174 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:51.174 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:51.174 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:51.174 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:35:51.175 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:51.175 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:51.175 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:51.175 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:51.175 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:51.176 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:51.176 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:51.176 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:51.176 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:51.176 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:51.177 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:35:51.177 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:51.177 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:35:51.177 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:35:51.177 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:35:51.178 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:51.178 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:35:51.179 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.007158_324.1 2021-09-17T10:35:56.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:56.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:35:56.181 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:35:56.224 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:35:56.224 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:35:56.343 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:35:56.344 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:35:56.344 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:35:56.344 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:35:56.345 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:35:56.345 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:35:56.345 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:56.346 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:56.346 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:35:56.346 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:56.346 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:35:56.347 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:56.347 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:56.347 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:35:56.347 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:56.348 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:35:56.348 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:56.348 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:35:56.348 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:56.349 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:56.349 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:56.349 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:56.349 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:35:56.349 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:56.350 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:56.350 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.001301_323.1", 2021-09-17T10:35:56.350 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:56.350 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:56.350 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:56.351 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.001301Z", 2021-09-17T10:35:56.351 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:56.351 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:56.351 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:56.352 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:56.352 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:35:56.352 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:56.352 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:56.352 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:56.353 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:56.353 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:56.353 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:56.353 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:56.354 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:56.354 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:56.354 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:56.355 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:35:56.355 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:35:56.355 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:35:56.355 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:35:56.356 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:35:56.356 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:35:56.356 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:35:56.356 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:35:56.357 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:35:56.357 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:35:56.357 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:56.357 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:56.358 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:35:56.358 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:35:56.358 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:35:56.358 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:35:56.359 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:35:56.359 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:35:56.359 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:35:56.360 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.001301_323.1 2021-09-17T10:36:01.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:01.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:01.363 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:36:01.430 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:36:01.431 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:36:01.857 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:36:01.857 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:36:01.857 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:36:01.857 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:36:01.858 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:36:01.858 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:36:01.858 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:01.858 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:01.859 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:36:01.859 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:01.859 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:36:01.859 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:01.860 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:01.860 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:36:01.860 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:01.860 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:36:01.861 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:01.861 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:36:01.861 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:01.861 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:01.862 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:01.862 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:01.862 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:36:01.862 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:01.863 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:01.863 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.249312_334.1", 2021-09-17T10:36:01.863 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:01.863 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:01.864 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:01.864 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.249312Z", 2021-09-17T10:36:01.864 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:01.864 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:01.864 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:01.865 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:01.865 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:36:01.865 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:01.865 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:01.865 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:01.866 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:01.866 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:01.866 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:01.867 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:01.867 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:01.867 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:01.867 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:01.867 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:01.868 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:36:01.868 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:01.868 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:01.868 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:01.869 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:01.869 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:01.869 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:01.869 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:01.869 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:01.870 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:01.870 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:01.870 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:36:01.870 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:01.871 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:36:01.871 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:36:01.871 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:36:01.871 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:01.872 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:36:01.873 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.249312_334.1 2021-09-17T10:36:06.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:06.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:06.875 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:36:06.932 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:36:06.932 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:36:07.029 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:36:07.029 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:36:07.030 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:36:07.030 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:36:07.030 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:36:07.030 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:36:07.031 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:07.031 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:07.031 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:36:07.031 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:07.032 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:36:07.032 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:07.032 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:07.032 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:36:07.032 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:07.033 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:36:07.033 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:07.033 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:36:07.033 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:07.033 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:07.034 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:07.034 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:07.034 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:36:07.034 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:07.034 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:07.035 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:36:07.035 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:07.035 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:07.035 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:07.035 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:36:07.036 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:07.036 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:07.036 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:07.036 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:07.036 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:36:07.037 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:07.037 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:07.037 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:07.037 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:07.038 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:07.038 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:07.038 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:07.038 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:07.038 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:07.039 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:07.039 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:07.039 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:36:07.039 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:07.039 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:07.040 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:07.040 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:07.040 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:07.040 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:07.041 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:07.041 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:07.041 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:07.041 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:07.041 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:36:07.042 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:07.042 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:36:07.042 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:36:07.042 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:36:07.042 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:07.043 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:36:07.044 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:36:12.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:12.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:12.046 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:36:12.089 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:36:12.089 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:36:12.202 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:36:12.203 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:36:12.203 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:36:12.203 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:36:12.203 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:36:12.204 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:36:12.204 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:12.204 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:12.204 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:36:12.204 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:12.205 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:36:12.205 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:12.205 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:12.205 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:36:12.206 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:12.206 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:36:12.206 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:12.206 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:36:12.207 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:12.207 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:12.207 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:12.207 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:12.208 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:36:12.208 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:12.208 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:12.208 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.959198_322.1", 2021-09-17T10:36:12.209 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:12.209 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:12.209 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:12.209 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.959198Z", 2021-09-17T10:36:12.209 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:12.210 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:12.210 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:12.210 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:12.210 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:36:12.211 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:12.211 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:12.211 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:12.211 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:12.212 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:12.212 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:12.212 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:12.212 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:12.213 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:12.213 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:12.213 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:12.213 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:36:12.214 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:12.214 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:12.214 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:12.214 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:12.214 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:12.215 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:12.215 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:12.215 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:12.215 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:12.216 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:12.216 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:36:12.216 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:12.216 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:36:12.217 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:36:12.217 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:36:12.217 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:12.217 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:36:12.218 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.959198_322.1 2021-09-17T10:36:17.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:17.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:17.221 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:36:17.309 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:36:17.309 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:36:17.685 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:36:17.685 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:36:17.685 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:36:17.686 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:36:17.686 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:36:17.686 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:36:17.687 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:17.687 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:17.687 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:36:17.687 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:17.687 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:36:17.688 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:17.688 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:17.688 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:36:17.688 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:17.689 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:36:17.689 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:17.689 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:36:17.689 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:17.689 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:17.690 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:17.690 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:17.690 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:36:17.690 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:17.691 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:17.691 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:36:17.691 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:17.691 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:17.691 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:17.692 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:36:17.692 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:17.692 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:17.692 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:17.693 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:17.693 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:36:17.693 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:17.693 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:17.693 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:17.694 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:17.694 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:17.694 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:17.694 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:17.695 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:17.695 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:17.695 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:17.695 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:17.695 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:36:17.696 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:17.696 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:17.696 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:17.696 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:17.696 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:17.697 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:17.697 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:17.697 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:17.697 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:17.698 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:17.698 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:36:17.698 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:17.699 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:36:17.699 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:36:17.699 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:36:17.699 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:17.699 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:36:17.701 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:36:22.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:22.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:22.703 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:36:22.749 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:36:22.749 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:36:22.877 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:36:22.877 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:36:22.878 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:36:22.878 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:36:22.878 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:36:22.879 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:36:22.879 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:22.879 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:22.879 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:36:22.880 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:22.880 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:36:22.880 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:22.880 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:22.881 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:36:22.881 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:22.881 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:36:22.881 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:22.881 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:36:22.882 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:22.882 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:22.882 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:22.882 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:22.883 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:36:22.883 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:22.883 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:22.883 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.001301_323.1", 2021-09-17T10:36:22.883 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:22.884 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:22.884 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:22.884 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.001301Z", 2021-09-17T10:36:22.884 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:22.884 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:22.885 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:22.885 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:22.885 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:36:22.885 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:22.886 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:22.886 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:22.886 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:22.886 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:22.886 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:22.887 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:22.887 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:22.887 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:22.887 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:22.888 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:22.888 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:36:22.888 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:22.888 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:22.889 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:22.889 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:22.889 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:22.889 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:22.890 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:22.890 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:22.890 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:22.890 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:22.890 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:36:22.891 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:22.891 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:36:22.891 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:36:22.891 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:36:22.891 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:22.892 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:36:22.893 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.001301_323.1 2021-09-17T10:36:27.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:27.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:27.896 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:36:27.938 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:36:27.939 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:36:28.070 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:36:28.071 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:36:28.071 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:36:28.071 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:36:28.071 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:36:28.071 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:36:28.072 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:28.072 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:28.072 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:36:28.072 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:28.073 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:36:28.073 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:28.073 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:28.073 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:36:28.074 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:28.074 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:36:28.074 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:28.074 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:36:28.075 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:28.075 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:28.075 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:28.075 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:28.075 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:36:28.076 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:28.076 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:28.076 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:36:28.076 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:28.077 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:28.077 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:28.077 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:36:28.077 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:28.077 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:28.078 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:28.078 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:28.078 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:36:28.078 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:28.079 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:28.079 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:28.079 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:28.079 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:28.080 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:28.080 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:28.080 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:28.080 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:28.080 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:28.081 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:28.081 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:36:28.081 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:28.081 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:28.082 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:28.082 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:28.082 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:28.082 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:28.082 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:28.083 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:28.083 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:28.083 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:28.083 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:36:28.083 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:28.084 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:36:28.084 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:36:28.084 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:36:28.084 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:28.085 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:36:28.086 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:36:33.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:33.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:33.088 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:36:33.202 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:36:33.202 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:36:33.604 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:36:33.604 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:36:33.605 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:36:33.605 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:36:33.605 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:36:33.605 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:36:33.606 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:33.606 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:33.606 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:36:33.606 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:33.607 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:36:33.607 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:33.607 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:33.607 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:36:33.607 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:33.608 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:36:33.608 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:33.608 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:36:33.608 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:33.609 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:33.609 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:33.609 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:33.609 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:36:33.609 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:33.610 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:33.610 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:36:33.610 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:33.610 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:33.611 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:33.611 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:36:33.611 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:33.611 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:33.611 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:33.612 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:33.612 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:36:33.612 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:33.612 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:33.613 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:33.613 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:33.613 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:33.613 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:33.613 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:33.614 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:33.614 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:33.614 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:33.614 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:33.614 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:36:33.615 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:33.615 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:33.615 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:33.615 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:33.616 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:33.616 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:33.616 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:33.616 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:33.616 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:33.617 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:33.617 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:36:33.617 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:33.617 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:36:33.618 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:36:33.618 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:36:33.618 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:33.618 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:36:33.619 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:36:38.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:38.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:38.622 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:36:38.664 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:36:38.664 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:36:38.751 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:36:38.751 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:36:38.752 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:36:38.752 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:36:38.752 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:36:38.752 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:36:38.753 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:38.753 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:38.753 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:36:38.753 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:38.754 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:36:38.754 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:38.754 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:38.754 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:36:38.754 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:38.755 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:36:38.755 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:38.755 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:36:38.755 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:38.756 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:38.756 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:38.756 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:38.756 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:36:38.756 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:38.757 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:38.757 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.001301_323.1", 2021-09-17T10:36:38.757 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:38.757 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:38.757 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:38.758 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.001301Z", 2021-09-17T10:36:38.758 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:38.758 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:38.758 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:38.759 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:38.759 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:36:38.759 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:38.759 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:38.760 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:38.760 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:38.760 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:38.760 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:38.760 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:38.761 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:38.761 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:38.761 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:38.761 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:38.762 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:36:38.762 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:38.762 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:38.762 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:38.763 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:38.763 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:38.763 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:38.763 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:38.764 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:38.764 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:38.764 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:38.764 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:36:38.764 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:38.765 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:36:38.765 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:36:38.765 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:36:38.765 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:38.766 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:36:38.767 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.001301_323.1 2021-09-17T10:36:43.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:43.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:43.769 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:36:43.810 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:36:43.811 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:36:43.936 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:36:43.936 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:36:43.936 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:36:43.937 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:36:43.937 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:36:43.937 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:36:43.937 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:43.938 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:43.938 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:36:43.938 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:43.938 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:36:43.939 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:43.939 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:43.939 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:36:43.939 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:43.939 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:36:43.940 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:43.940 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:36:43.940 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:43.940 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:43.940 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:43.941 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:43.941 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:36:43.941 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:43.941 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:43.942 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:36:43.942 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:43.942 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:43.942 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:43.943 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:36:43.943 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:43.943 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:43.943 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:43.943 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:43.944 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:36:43.944 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:43.944 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:43.944 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:43.945 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:43.945 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:43.945 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:43.945 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:43.946 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:43.946 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:43.946 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:43.946 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:43.947 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:36:43.947 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:43.947 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:43.947 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:43.947 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:43.948 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:43.948 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:43.948 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:43.948 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:43.949 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:43.949 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:43.949 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:36:43.949 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:43.949 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:36:43.950 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:36:43.950 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:36:43.950 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:43.950 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:36:43.951 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:36:48.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:48.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:48.955 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:36:49.067 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:36:49.067 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:36:49.496 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:36:49.497 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:36:49.497 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:36:49.497 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:36:49.498 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:36:49.498 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:36:49.498 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:49.498 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:49.498 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:36:49.499 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:49.499 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:36:49.499 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:49.499 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:49.500 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:36:49.500 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:49.500 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:36:49.500 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:49.500 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:36:49.501 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:49.501 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:49.501 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:49.501 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:49.502 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:36:49.502 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:49.502 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:49.502 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:36:49.502 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:49.503 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:49.503 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:49.503 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:36:49.503 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:49.503 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:49.504 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:49.504 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:49.504 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:36:49.504 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:49.505 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:49.505 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:49.505 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:49.505 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:49.505 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:49.506 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:49.506 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:49.506 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:49.506 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:49.506 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:49.507 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:36:49.507 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:49.507 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:49.507 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:49.508 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:49.508 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:49.508 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:49.508 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:49.508 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:49.509 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:49.509 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:49.509 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:36:49.509 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:49.510 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:36:49.510 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:36:49.510 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:36:49.510 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:49.510 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:36:49.512 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:36:54.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:54.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:54.514 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:36:54.556 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:36:54.556 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:36:54.683 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:36:54.684 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:36:54.684 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:36:54.684 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:36:54.684 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:36:54.684 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:36:54.685 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:54.685 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:54.685 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:36:54.685 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:54.685 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:36:54.686 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:54.686 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:54.686 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:36:54.686 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:54.687 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:36:54.687 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:54.687 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:36:54.687 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:54.687 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:54.688 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:54.688 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:54.688 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:36:54.688 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:54.688 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:54.689 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:36:54.689 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:54.689 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:54.689 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:54.689 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:36:54.690 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:54.690 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:54.690 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:54.690 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:54.690 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:36:54.691 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:54.691 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:54.691 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:54.691 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:54.692 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:54.692 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:54.692 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:54.692 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:54.692 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:54.693 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:54.693 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:36:54.693 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:36:54.693 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:36:54.694 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:36:54.694 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:36:54.694 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:36:54.694 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:36:54.694 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:36:54.695 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:36:54.695 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:36:54.695 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:54.695 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:54.695 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:36:54.696 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:36:54.696 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:36:54.696 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:36:54.696 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:36:54.696 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:36:54.697 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:36:54.698 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:36:59.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:59.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:36:59.701 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:36:59.773 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:36:59.773 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:00.229 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:37:00.230 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:37:00.230 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:37:00.230 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:37:00.230 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:37:00.231 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:00.231 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:00.231 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:00.231 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:37:00.232 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:00.232 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:37:00.232 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:00.232 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:00.232 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:00.233 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:00.233 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:37:00.233 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:00.233 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:00.233 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:00.234 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:00.234 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:00.234 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:00.234 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:37:00.235 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:00.235 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:00.235 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.673300_353.1", 2021-09-17T10:37:00.235 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:00.235 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:00.236 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:00.236 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.673300Z", 2021-09-17T10:37:00.236 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:00.236 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:00.236 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:00.237 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:00.237 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:37:00.237 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:00.237 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:00.238 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:00.238 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:00.238 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:00.238 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:00.238 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:00.239 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:00.239 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:00.239 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:00.239 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:00.240 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:37:00.240 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:00.240 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:00.240 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:00.240 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:00.241 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:00.241 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:00.241 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:00.241 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:00.241 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:00.242 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:00.242 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:37:00.242 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:00.242 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:37:00.243 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:37:00.243 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:37:00.243 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:00.243 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:37:00.244 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.673300_353.1 2021-09-17T10:37:05.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:05.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:05.247 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:05.295 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:05.295 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:05.405 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:37:05.405 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:37:05.406 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:37:05.406 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:37:05.406 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:37:05.407 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:05.407 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:05.407 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:05.407 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:37:05.408 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:05.408 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:37:05.408 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:05.408 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:05.409 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:05.409 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:05.409 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:37:05.409 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:05.410 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:05.410 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:05.410 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:05.410 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:05.411 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:05.411 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:37:05.411 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:05.411 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:05.412 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:37:05.412 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:05.412 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:05.412 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:05.413 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:37:05.413 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:05.413 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:05.413 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:05.413 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:05.414 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:37:05.414 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:05.414 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:05.414 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:05.415 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:05.415 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:05.415 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:05.415 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:05.416 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:05.416 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:05.416 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:05.416 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:05.416 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:37:05.417 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:05.417 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:05.417 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:05.417 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:05.418 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:05.418 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:05.418 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:05.418 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:05.419 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:05.419 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:05.419 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:37:05.419 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:05.420 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:37:05.420 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:37:05.420 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:37:05.420 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:05.420 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:37:05.422 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:37:10.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:10.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:10.425 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:10.468 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:10.468 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:10.585 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:37:10.585 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:37:10.585 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:37:10.585 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:37:10.586 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:37:10.586 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:10.586 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:10.586 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:10.587 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:37:10.587 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:10.587 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:37:10.587 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:10.587 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:10.588 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:10.588 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:10.588 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:37:10.588 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:10.588 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:10.589 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:10.589 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:10.589 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:10.589 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:10.589 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:37:10.590 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:10.590 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:10.590 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.959198_322.1", 2021-09-17T10:37:10.590 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:10.590 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:10.591 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:10.591 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.959198Z", 2021-09-17T10:37:10.591 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:10.591 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:10.591 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:10.592 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:10.592 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:37:10.592 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:10.593 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:10.593 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:10.593 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:10.593 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:10.593 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:10.594 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:10.594 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:10.594 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:10.594 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:10.594 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:10.595 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:37:10.595 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:10.595 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:10.595 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:10.595 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:10.596 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:10.596 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:10.596 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:10.596 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:10.596 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:10.597 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:10.597 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:37:10.597 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:10.597 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:37:10.597 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:37:10.598 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:37:10.598 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:10.598 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:37:10.599 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.959198_322.1 2021-09-17T10:37:15.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:15.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:15.602 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:15.672 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:15.673 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:16.039 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:37:16.040 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:37:16.040 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:37:16.040 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:37:16.040 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:37:16.040 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:16.041 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:16.041 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:16.041 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:37:16.041 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:16.041 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:37:16.042 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:16.042 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:16.042 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:16.042 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:16.042 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:37:16.043 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:16.043 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:16.043 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:16.043 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:16.043 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:16.044 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:16.044 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:37:16.044 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:16.044 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:16.044 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:37:16.045 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:16.045 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:16.045 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:16.045 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:37:16.046 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:16.046 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:16.046 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:16.046 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:16.046 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:37:16.047 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:16.047 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:16.047 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:16.047 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:16.047 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:16.048 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:16.048 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:16.048 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:16.048 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:16.048 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:16.049 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:16.049 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:37:16.049 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:16.049 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:16.049 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:16.050 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:16.050 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:16.050 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:16.050 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:16.050 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:16.051 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:16.051 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:16.051 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:37:16.051 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:16.052 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:37:16.052 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:37:16.052 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:37:16.052 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:16.052 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:37:16.053 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:37:21.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:21.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:21.056 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:21.093 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:21.093 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:21.202 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:37:21.203 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:37:21.203 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:37:21.203 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:37:21.203 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:37:21.204 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:21.204 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:21.204 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:21.204 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:37:21.204 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:21.205 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:37:21.205 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:21.205 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:21.205 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:21.206 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:21.206 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:37:21.206 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:21.206 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:21.206 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:21.207 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:21.207 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:21.207 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:21.207 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:37:21.208 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:21.208 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:21.208 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:37:21.208 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:21.208 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:21.209 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:21.209 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:37:21.209 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:21.209 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:21.210 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:21.210 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:21.210 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:37:21.210 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:21.210 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:21.211 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:21.211 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:21.211 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:21.211 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:21.212 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:21.212 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:21.212 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:21.212 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:21.212 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:21.213 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:37:21.213 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:21.213 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:21.213 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:21.214 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:21.214 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:21.214 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:21.214 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:21.214 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:21.215 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:21.215 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:21.215 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:37:21.215 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:21.216 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:37:21.216 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:37:21.216 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:37:21.216 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:21.216 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:37:21.217 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:37:26.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:26.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:26.220 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:26.264 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:26.265 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:26.384 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:37:26.384 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:37:26.384 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:37:26.385 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:37:26.385 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:37:26.385 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:26.385 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:26.386 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:26.386 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:37:26.386 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:26.386 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:37:26.387 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:26.387 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:26.387 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:26.387 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:26.388 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:37:26.388 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:26.388 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:26.388 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:26.388 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:26.389 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:26.389 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:26.389 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:37:26.389 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:26.390 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:26.390 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.925654_356.1", 2021-09-17T10:37:26.390 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:26.390 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:26.390 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:26.391 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.925654Z", 2021-09-17T10:37:26.391 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:26.391 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:26.391 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:26.391 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:26.392 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:37:26.392 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:26.392 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:26.392 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:26.393 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:26.393 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:26.393 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:26.393 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:26.393 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:26.394 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:26.394 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:26.394 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:26.394 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:37:26.395 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:26.395 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:26.395 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:26.395 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:26.395 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:26.396 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:26.396 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:26.396 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:26.396 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:26.397 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:26.397 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:37:26.397 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:26.397 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:37:26.397 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:37:26.398 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:37:26.398 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:26.398 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:37:26.399 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.925654_356.1 2021-09-17T10:37:31.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:31.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:31.402 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:31.489 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:31.489 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:31.877 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:37:31.878 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:37:31.878 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:37:31.878 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:37:31.878 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:37:31.879 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:31.879 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:31.879 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:31.879 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:37:31.880 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:31.880 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:37:31.880 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:31.880 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:31.880 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:31.881 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:31.881 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:37:31.881 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:31.881 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:31.882 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:31.882 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:31.882 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:31.882 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:31.883 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:37:31.883 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:31.883 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:31.883 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.256150_335.1", 2021-09-17T10:37:31.883 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:31.884 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:31.884 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:31.884 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.256150Z", 2021-09-17T10:37:31.884 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:31.885 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:31.885 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:31.885 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:31.885 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:37:31.885 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:31.886 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:31.886 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:31.886 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:31.886 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:31.887 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:31.887 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:31.887 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:31.887 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:31.887 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:31.888 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:31.888 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:37:31.888 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:31.888 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:31.888 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:31.889 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:31.889 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:31.889 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:31.889 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:31.890 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:31.890 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:31.890 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:31.890 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:37:31.890 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:31.891 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:37:31.891 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:37:31.891 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:37:31.891 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:31.892 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:37:31.893 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.256150_335.1 2021-09-17T10:37:36.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:36.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:36.896 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:36.931 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:36.932 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:37.042 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:37:37.043 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:37:37.043 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:37:37.043 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:37:37.043 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:37:37.044 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:37.044 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:37.044 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:37.044 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:37:37.044 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:37.045 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:37:37.045 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:37.045 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:37.045 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:37.045 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:37.046 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:37:37.046 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:37.046 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:37.046 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:37.047 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:37.047 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:37.047 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:37.047 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:37:37.047 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:37.048 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:37.048 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:37:37.048 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:37.048 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:37.048 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:37.049 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:37:37.049 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:37.049 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:37.049 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:37.049 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:37.050 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:37:37.050 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:37.050 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:37.050 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:37.050 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:37.051 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:37.051 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:37.051 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:37.051 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:37.051 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:37.052 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:37.052 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:37.052 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:37:37.052 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:37.052 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:37.053 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:37.053 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:37.053 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:37.053 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:37.053 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:37.054 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:37.054 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:37.054 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:37.054 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:37:37.054 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:37.055 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:37:37.055 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:37:37.055 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:37:37.055 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:37.055 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:37:37.056 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:37:42.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:42.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:42.059 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:42.100 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:42.101 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:42.221 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:37:42.221 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:37:42.221 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:37:42.221 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:37:42.221 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:37:42.222 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:42.222 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:42.222 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:42.222 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:37:42.223 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:42.223 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:37:42.223 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:42.223 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:42.223 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:42.224 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:42.224 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:37:42.224 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:42.224 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:42.225 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:42.225 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:42.225 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:42.225 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:42.225 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:37:42.226 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:42.226 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:42.226 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:37:42.226 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:42.226 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:42.227 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:42.227 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:37:42.227 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:42.227 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:42.228 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:42.228 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:42.228 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:37:42.228 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:42.228 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:42.229 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:42.229 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:42.229 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:42.229 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:42.229 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:42.230 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:42.230 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:42.230 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:42.230 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:42.231 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:37:42.231 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:42.231 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:42.231 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:42.231 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:42.232 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:42.232 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:42.232 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:42.232 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:42.232 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:42.233 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:42.233 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:37:42.233 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:42.233 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:37:42.234 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:37:42.234 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:37:42.234 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:42.234 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:37:42.235 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:37:47.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:47.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:47.238 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:47.361 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:47.361 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:47.856 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:37:47.856 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:37:47.857 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:37:47.857 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:37:47.857 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:37:47.857 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:47.858 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:47.858 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:47.858 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:37:47.858 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:47.859 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:37:47.859 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:47.859 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:47.859 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:47.859 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:47.860 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:37:47.860 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:47.860 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:47.860 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:47.861 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:47.861 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:47.861 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:47.861 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:37:47.861 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:47.862 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:47.862 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:37:47.862 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:47.862 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:47.862 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:47.863 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:37:47.863 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:47.863 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:47.863 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:47.864 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:47.864 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:37:47.864 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:47.864 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:47.864 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:47.865 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:47.865 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:47.865 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:47.865 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:47.865 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:47.866 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:47.866 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:47.866 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:47.866 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:37:47.866 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:47.867 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:47.867 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:47.867 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:47.867 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:47.867 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:47.868 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:47.868 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:47.868 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:47.868 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:47.868 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:37:47.869 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:47.869 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:37:47.869 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:37:47.869 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:37:47.870 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:47.870 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:37:47.871 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:37:52.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2021-09-17T10:37:52.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:52.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:52.876 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:52.918 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:52.918 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:52.980 INFO:teuthology.orchestra.run.smithi018.stderr:No --period given, using current period=3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac 2021-09-17T10:37:53.005 INFO:teuthology.orchestra.run.smithi018.stdout:[ 2021-09-17T10:37:53.005 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:53.005 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:53.006 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:11:08.462214Z" 2021-09-17T10:37:53.006 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:53.006 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:53.006 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631874691.729253_479.1", 2021-09-17T10:37:53.006 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "2021-09-17T10:31:31.729253Z" 2021-09-17T10:37:53.007 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:53.007 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:53.007 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:53.007 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:37:53.008 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:53.008 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:53.008 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:53.008 INFO:teuthology.orchestra.run.smithi018.stdout: "last_update": "0.000000" 2021-09-17T10:37:53.009 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:53.009 INFO:teuthology.orchestra.run.smithi018.stdout:] 2021-09-17T10:37:53.010 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-17T10:37:53.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:53.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:53.012 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:53.056 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:53.056 INFO:teuthology.orchestra.run.smithi050.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:53.171 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2021-09-17T10:37:53.172 INFO:teuthology.orchestra.run.smithi050.stdout: "sync_status": { 2021-09-17T10:37:53.172 INFO:teuthology.orchestra.run.smithi050.stdout: "info": { 2021-09-17T10:37:53.172 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "sync", 2021-09-17T10:37:53.172 INFO:teuthology.orchestra.run.smithi050.stdout: "num_shards": 4, 2021-09-17T10:37:53.173 INFO:teuthology.orchestra.run.smithi050.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:53.173 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:37:53.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:37:53.173 INFO:teuthology.orchestra.run.smithi050.stdout: "markers": [ 2021-09-17T10:37:53.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:37:53.174 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 0, 2021-09-17T10:37:53.174 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:37:53.174 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:37:53.174 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:53.175 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:37:53.175 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 1, 2021-09-17T10:37:53.175 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:37:53.175 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:53.176 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:37:53.176 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:37:53.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:37:53.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:37:53.176 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 1, 2021-09-17T10:37:53.177 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:37:53.177 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:37:53.177 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "1_1631874691.729253_479.1", 2021-09-17T10:37:53.177 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:37:53.178 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:37:53.178 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:37:53.178 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "2021-09-17T10:31:31.729253Z", 2021-09-17T10:37:53.179 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:37:53.179 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:37:53.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:37:53.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:37:53.180 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 2, 2021-09-17T10:37:53.180 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:37:53.180 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:37:53.180 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:37:53.180 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:37:53.181 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:37:53.181 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:37:53.181 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:37:53.181 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:37:53.182 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:37:53.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:37:53.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2021-09-17T10:37:53.182 INFO:teuthology.orchestra.run.smithi050.stdout: "key": 3, 2021-09-17T10:37:53.183 INFO:teuthology.orchestra.run.smithi050.stdout: "val": { 2021-09-17T10:37:53.183 INFO:teuthology.orchestra.run.smithi050.stdout: "state": 1, 2021-09-17T10:37:53.183 INFO:teuthology.orchestra.run.smithi050.stdout: "marker": "", 2021-09-17T10:37:53.183 INFO:teuthology.orchestra.run.smithi050.stdout: "next_step_marker": "", 2021-09-17T10:37:53.183 INFO:teuthology.orchestra.run.smithi050.stdout: "total_entries": 0, 2021-09-17T10:37:53.184 INFO:teuthology.orchestra.run.smithi050.stdout: "pos": 0, 2021-09-17T10:37:53.184 INFO:teuthology.orchestra.run.smithi050.stdout: "timestamp": "0.000000", 2021-09-17T10:37:53.184 INFO:teuthology.orchestra.run.smithi050.stdout: "realm_epoch": 2 2021-09-17T10:37:53.184 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:37:53.185 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:37:53.185 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2021-09-17T10:37:53.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2021-09-17T10:37:53.185 INFO:teuthology.orchestra.run.smithi050.stdout: "full_sync": { 2021-09-17T10:37:53.185 INFO:teuthology.orchestra.run.smithi050.stdout: "total": 1, 2021-09-17T10:37:53.186 INFO:teuthology.orchestra.run.smithi050.stdout: "complete": 1 2021-09-17T10:37:53.186 INFO:teuthology.orchestra.run.smithi050.stdout: } 2021-09-17T10:37:53.186 INFO:teuthology.orchestra.run.smithi050.stdout:} 2021-09-17T10:37:53.188 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-17T10:37:53.188 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-17T10:37:53.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:53.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:53.190 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:53.232 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:53.232 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:53.355 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:37:53.355 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:37:53.356 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:37:53.356 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:37:53.356 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:37:53.356 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:53.357 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:53.357 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:53.357 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:37:53.357 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:53.357 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:37:53.358 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:53.358 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:53.358 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:53.358 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:53.359 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:37:53.359 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:53.359 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:53.359 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:53.360 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:53.360 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:53.360 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:53.360 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:37:53.361 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:53.361 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:53.361 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.038405_327.1", 2021-09-17T10:37:53.361 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:53.361 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:53.362 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:53.362 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.038405Z", 2021-09-17T10:37:53.362 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:53.362 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:53.363 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:53.363 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:53.363 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:37:53.363 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:53.364 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:53.364 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:53.364 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:53.364 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:53.364 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:53.365 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:53.365 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:53.365 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:53.365 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:53.366 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:53.366 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:37:53.366 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:53.366 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:53.367 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:53.367 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:53.367 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:53.367 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:53.367 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:53.368 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:53.368 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:53.368 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:53.368 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:37:53.369 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:53.369 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:37:53.369 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:37:53.369 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:37:53.370 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:53.370 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:37:53.371 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.038405_327.1 2021-09-17T10:37:58.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:58.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:37:58.373 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:37:58.414 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:37:58.414 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:37:58.604 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:37:58.605 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:37:58.605 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:37:58.605 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:37:58.605 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:37:58.606 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:37:58.606 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:58.606 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:58.606 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:37:58.607 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:58.607 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:37:58.607 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:58.607 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:58.607 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:37:58.608 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:58.608 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:37:58.608 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:58.608 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:37:58.609 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:58.609 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:58.609 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:58.609 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:58.610 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:37:58.610 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:58.610 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:58.610 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.334371_338.1", 2021-09-17T10:37:58.610 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:58.611 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:58.611 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:58.611 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.334371Z", 2021-09-17T10:37:58.611 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:58.612 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:58.612 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:58.612 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:58.612 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:37:58.612 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:58.613 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:58.613 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:58.613 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:58.613 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:58.613 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:58.614 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:58.614 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:58.614 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:58.614 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:58.614 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:37:58.615 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:37:58.615 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:37:58.615 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:37:58.615 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:37:58.615 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:37:58.616 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:37:58.616 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:37:58.616 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:37:58.616 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:37:58.616 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:58.617 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:58.617 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:37:58.617 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:37:58.617 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:37:58.617 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:37:58.618 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:37:58.618 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:37:58.618 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:37:58.619 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.334371_338.1 2021-09-17T10:38:03.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:03.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:03.622 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:38:03.714 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:38:03.714 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:38:03.906 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:38:03.906 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:38:03.907 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:38:03.907 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:38:03.907 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:38:03.907 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:38:03.908 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:03.908 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:03.908 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:38:03.908 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:03.908 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:38:03.909 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:03.909 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:03.909 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:38:03.909 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:03.910 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:38:03.910 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:03.910 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:38:03.910 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:03.911 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:03.911 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:03.911 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:03.911 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:38:03.911 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:03.912 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:03.912 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.038405_327.1", 2021-09-17T10:38:03.912 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:03.912 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:03.912 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:03.913 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.038405Z", 2021-09-17T10:38:03.913 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:03.913 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:03.913 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:03.913 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:03.914 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:38:03.914 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:03.914 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:03.914 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:03.915 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:03.915 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:03.915 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:03.915 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:03.915 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:03.916 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:03.916 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:03.916 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:03.916 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:38:03.916 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:03.917 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:03.917 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:03.917 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:03.917 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:03.917 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:03.918 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:03.918 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:03.918 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:03.918 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:03.918 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:38:03.919 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:03.919 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:38:03.919 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:38:03.919 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:38:03.919 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:03.920 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:38:03.921 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.038405_327.1 2021-09-17T10:38:08.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:08.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:08.923 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:38:08.965 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:38:08.965 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:38:09.085 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:38:09.086 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:38:09.086 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:38:09.086 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:38:09.087 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:38:09.087 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:38:09.087 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:09.087 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:09.088 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:38:09.088 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:09.088 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:38:09.088 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:09.088 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:09.089 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:38:09.089 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:09.089 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:38:09.089 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:09.090 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:38:09.090 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:09.090 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:09.090 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:09.090 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:09.091 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:38:09.091 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:09.091 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:09.091 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:38:09.092 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:09.092 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:09.092 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:09.092 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:38:09.092 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:09.093 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:09.093 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:09.093 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:09.093 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:38:09.094 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:09.094 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:09.094 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:09.094 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:09.095 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:09.095 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:09.095 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:09.095 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:09.095 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:09.096 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:09.096 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:09.096 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:38:09.096 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:09.097 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:09.097 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:09.097 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:09.097 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:09.097 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:09.098 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:09.098 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:09.098 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:09.098 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:09.099 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:38:09.099 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:09.099 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:38:09.100 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:38:09.100 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:38:09.100 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:09.100 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:38:09.101 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.952206_320.1 2021-09-17T10:38:14.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:14.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:14.104 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:38:14.200 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:38:14.200 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:38:14.593 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:38:14.593 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:38:14.593 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:38:14.593 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:38:14.594 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:38:14.594 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:38:14.594 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:14.594 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:14.595 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:38:14.595 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:14.595 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:38:14.595 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:14.596 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:14.596 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:38:14.596 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:14.596 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:38:14.596 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:14.597 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:38:14.597 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:14.597 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:14.597 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:14.598 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:14.598 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:38:14.598 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:14.598 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:14.599 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:38:14.599 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:14.599 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:14.599 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:14.600 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:38:14.600 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:14.600 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:14.600 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:14.600 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:14.601 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:38:14.601 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:14.601 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:14.601 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:14.602 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:14.602 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:14.602 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:14.603 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:14.603 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:14.603 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:14.603 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:14.603 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:14.604 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:38:14.604 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:14.604 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:14.604 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:14.604 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:14.605 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:14.605 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:14.605 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:14.605 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:14.606 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:14.606 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:14.606 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:38:14.606 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:14.606 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:38:14.607 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:38:14.607 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:38:14.607 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:14.608 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:38:14.609 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:38:19.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:19.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:19.612 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:38:19.655 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:38:19.655 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:38:19.776 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:38:19.776 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:38:19.776 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:38:19.776 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:38:19.777 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:38:19.777 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:38:19.777 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:19.777 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:19.778 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:38:19.778 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:19.778 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:38:19.778 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:19.779 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:19.779 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:38:19.779 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:19.779 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:38:19.779 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:19.780 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:38:19.780 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:19.780 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:19.780 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:19.781 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:19.781 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:38:19.781 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:19.781 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:19.782 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.044111_328.1", 2021-09-17T10:38:19.782 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:19.782 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:19.782 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:19.782 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.044111Z", 2021-09-17T10:38:19.783 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:19.783 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:19.783 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:19.783 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:19.784 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:38:19.784 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:19.784 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:19.784 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:19.785 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:19.785 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:19.785 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:19.785 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:19.785 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:19.786 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:19.786 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:19.786 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:19.786 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:38:19.787 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:19.787 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:19.787 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:19.787 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:19.787 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:19.788 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:19.788 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:19.788 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:19.788 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:19.789 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:19.789 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:38:19.789 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:19.790 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:38:19.790 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:38:19.790 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:38:19.790 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:19.791 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:38:19.792 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.044111_328.1 2021-09-17T10:38:24.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:24.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:24.795 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:38:24.836 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:38:24.836 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:38:25.148 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:38:25.148 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:38:25.148 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:38:25.148 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:38:25.149 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:38:25.149 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:38:25.149 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:25.149 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:25.150 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:38:25.150 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:25.150 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:38:25.150 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:25.150 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:25.151 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:38:25.151 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:25.151 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:38:25.151 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:25.152 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:38:25.152 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:25.152 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:25.152 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:25.153 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:25.153 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:38:25.153 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:25.153 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:25.153 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.038405_327.1", 2021-09-17T10:38:25.154 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:25.154 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:25.154 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:25.154 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.038405Z", 2021-09-17T10:38:25.155 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:25.155 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:25.155 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:25.155 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:25.155 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:38:25.156 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:25.156 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:25.156 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:25.156 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:25.157 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:25.157 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:25.157 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:25.157 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:25.158 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:25.158 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:25.158 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:25.158 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:38:25.159 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:25.159 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:25.159 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:25.159 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:25.159 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:25.160 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:25.160 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:25.160 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:25.160 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:25.161 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:25.161 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:38:25.161 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:25.161 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:38:25.161 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:38:25.162 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:38:25.162 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:25.162 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:38:25.163 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.038405_327.1 2021-09-17T10:38:30.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:30.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:30.166 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:38:30.254 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:38:30.254 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:38:30.456 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:38:30.457 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:38:30.457 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:38:30.457 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:38:30.458 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:38:30.458 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:38:30.458 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:30.458 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:30.458 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:38:30.459 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:30.459 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:38:30.459 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:30.459 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:30.459 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:38:30.460 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:30.460 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:38:30.460 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:30.460 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:38:30.460 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:30.461 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:30.461 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:30.461 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:30.461 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:38:30.461 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:30.462 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:30.462 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:38:30.462 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:30.462 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:30.463 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:30.463 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:38:30.463 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:30.463 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:30.463 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:30.464 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:30.464 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:38:30.464 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:30.465 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:30.465 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:30.465 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:30.466 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:30.466 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:30.466 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:30.467 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:30.467 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:30.467 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:30.467 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:30.467 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:38:30.468 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:30.468 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:30.468 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:30.468 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:30.468 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:30.469 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:30.469 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:30.469 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:30.469 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:30.470 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:30.470 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:38:30.470 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:30.470 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:38:30.470 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:38:30.471 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:38:30.471 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:30.471 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:38:30.472 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:38:35.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:35.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:35.475 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:38:35.516 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:38:35.517 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:38:35.636 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:38:35.637 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:38:35.637 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:38:35.637 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:38:35.638 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:38:35.638 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:38:35.638 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:35.639 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:35.639 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:38:35.639 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:35.639 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:38:35.640 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:35.640 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:35.640 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:38:35.640 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:35.641 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:38:35.641 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:35.641 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:38:35.641 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:35.642 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:35.642 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:35.642 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:35.642 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:38:35.643 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:35.644 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:35.644 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.038405_327.1", 2021-09-17T10:38:35.644 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:35.644 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:35.645 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:35.645 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.038405Z", 2021-09-17T10:38:35.645 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:35.645 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:35.645 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:35.646 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:35.646 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:38:35.646 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:35.647 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:35.647 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:35.647 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:35.647 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:35.648 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:35.648 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:35.648 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:35.648 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:35.649 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:35.649 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:35.649 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:38:35.650 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:35.650 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:35.650 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:35.650 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:35.651 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:35.651 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:35.652 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:35.652 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:35.652 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:35.652 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:35.653 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:38:35.653 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:35.653 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:38:35.653 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:38:35.654 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:38:35.654 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:35.654 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:38:35.655 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.038405_327.1 2021-09-17T10:38:40.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:40.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:40.658 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:38:40.721 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:38:40.722 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:38:41.136 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:38:41.136 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:38:41.137 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:38:41.137 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:38:41.137 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:38:41.137 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:38:41.137 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:41.138 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:41.138 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:38:41.138 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:41.138 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:38:41.139 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:41.139 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:41.139 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:38:41.139 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:41.139 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:38:41.140 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:41.140 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:38:41.140 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:41.141 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:41.141 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:41.141 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:41.141 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:38:41.141 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:41.142 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:41.142 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:38:41.142 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:41.142 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:41.143 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:41.143 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:38:41.143 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:41.143 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:41.144 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:41.144 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:41.144 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:38:41.144 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:41.144 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:41.145 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:41.145 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:41.145 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:41.145 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:41.145 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:41.146 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:41.146 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:41.146 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:41.146 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:41.147 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:38:41.147 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:41.147 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:41.147 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:41.148 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:41.148 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:41.148 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:41.148 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:41.148 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:41.149 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:41.149 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:41.149 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:38:41.149 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:41.149 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:38:41.150 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:38:41.150 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:38:41.150 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:41.150 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:38:41.151 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:38:46.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:46.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:46.153 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:38:46.191 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:38:46.192 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:38:46.294 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:38:46.295 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:38:46.295 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:38:46.295 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:38:46.296 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:38:46.296 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:38:46.297 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:46.297 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:46.297 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:38:46.297 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:46.298 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:38:46.298 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:46.298 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:46.298 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:38:46.299 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:46.299 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:38:46.299 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:46.299 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:38:46.300 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:46.300 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:46.300 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:46.300 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:46.300 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:38:46.301 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:46.301 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:46.301 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.959198_322.1", 2021-09-17T10:38:46.301 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:46.302 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:46.302 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:46.302 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.959198Z", 2021-09-17T10:38:46.303 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:46.303 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:46.303 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:46.304 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:46.304 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:38:46.304 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:46.305 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:46.305 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:46.306 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:46.306 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:46.307 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:46.307 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:46.307 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:46.308 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:46.308 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:46.308 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:46.308 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:38:46.309 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:46.309 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:46.310 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:46.310 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:46.310 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:46.310 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:46.311 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:46.311 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:46.311 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:46.311 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:46.311 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:38:46.312 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:46.312 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:38:46.312 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:38:46.312 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:38:46.312 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:46.313 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:38:46.314 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.959198_322.1 2021-09-17T10:38:51.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:51.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:51.316 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:38:51.358 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:38:51.358 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:38:51.484 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:38:51.484 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:38:51.484 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:38:51.485 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:38:51.485 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:38:51.485 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:38:51.485 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:51.485 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:51.486 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:38:51.486 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:51.486 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:38:51.486 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:51.487 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:51.487 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:38:51.487 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:51.487 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:38:51.488 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:51.488 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:38:51.489 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:51.489 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:51.489 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:51.489 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:51.490 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:38:51.490 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:51.491 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:51.491 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.406824_341.1", 2021-09-17T10:38:51.492 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:51.492 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:51.492 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:51.493 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.406824Z", 2021-09-17T10:38:51.493 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:51.493 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:51.494 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:51.494 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:51.494 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:38:51.495 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:51.495 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:51.495 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:51.496 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:51.496 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:51.496 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:51.496 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:51.497 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:51.497 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:51.497 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:51.497 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:51.498 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:38:51.498 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:51.498 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:51.499 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:51.499 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:51.499 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:51.499 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:51.500 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:51.500 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:51.500 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:51.500 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:51.501 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:38:51.501 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:51.501 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:38:51.501 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:38:51.502 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:38:51.502 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:51.502 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:38:51.503 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.406824_341.1 2021-09-17T10:38:56.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:56.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:38:56.506 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:38:56.591 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:38:56.592 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:38:56.869 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:38:56.870 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:38:56.870 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:38:56.870 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:38:56.870 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:38:56.871 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:38:56.871 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:56.871 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:56.871 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:38:56.872 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:56.872 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:38:56.872 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:56.872 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:56.873 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:38:56.873 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:56.873 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:38:56.873 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:56.874 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:38:56.874 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:56.874 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:56.874 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:56.875 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:56.875 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:38:56.875 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:56.875 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:56.876 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:38:56.876 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:56.876 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:56.876 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:56.876 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:38:56.877 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:56.877 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:56.877 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:56.877 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:56.877 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:38:56.878 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:56.878 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:56.878 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:56.878 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:56.879 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:56.879 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:56.879 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:56.879 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:56.879 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:56.880 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:56.880 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:38:56.880 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:38:56.880 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:38:56.881 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:38:56.881 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:38:56.881 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:38:56.881 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:38:56.881 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:38:56.882 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:38:56.882 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:38:56.882 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:56.882 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:56.883 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:38:56.883 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:38:56.883 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:38:56.883 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:38:56.883 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:38:56.884 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:38:56.884 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:38:56.885 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:39:01.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:01.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:01.889 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:39:01.931 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:39:01.931 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:39:02.054 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:39:02.054 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:39:02.055 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:39:02.055 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:39:02.055 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:39:02.055 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:39:02.056 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:02.056 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:02.056 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:39:02.056 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:02.057 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:39:02.057 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:02.057 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:02.057 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:39:02.058 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:02.058 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:39:02.058 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:02.058 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:39:02.058 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:02.059 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:02.059 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:02.059 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:02.059 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:39:02.060 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:02.060 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:02.060 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:39:02.060 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:02.061 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:02.061 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:02.061 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:39:02.061 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:02.061 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:02.062 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:02.062 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:02.062 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:39:02.062 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:02.063 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:02.063 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:02.063 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:02.063 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:02.064 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:02.064 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:02.064 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:02.064 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:02.064 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:02.065 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:02.065 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:39:02.065 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:02.065 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:02.065 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:02.066 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:02.066 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:02.066 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:02.066 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:02.067 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:02.067 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:02.067 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:02.067 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:39:02.067 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:02.068 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:39:02.068 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:39:02.068 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:39:02.068 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:02.068 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:39:02.070 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:39:07.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:07.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:07.073 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:39:07.244 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:39:07.244 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:39:07.755 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:39:07.755 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:39:07.756 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:39:07.756 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:39:07.756 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:39:07.757 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:39:07.757 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:07.758 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:07.758 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:39:07.758 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:07.759 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:39:07.759 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:07.760 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:07.760 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:39:07.760 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:07.761 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:39:07.761 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:07.761 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:39:07.762 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:07.762 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:07.762 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:07.763 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:07.763 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:39:07.763 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:07.763 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:07.765 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.164606_332.1", 2021-09-17T10:39:07.765 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:07.765 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:07.765 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:07.766 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.164606Z", 2021-09-17T10:39:07.766 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:07.766 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:07.766 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:07.767 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:07.767 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:39:07.767 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:07.768 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:07.768 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:07.768 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:07.768 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:07.768 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:07.769 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:07.769 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:07.769 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:07.769 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:07.770 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:07.770 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:39:07.770 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:07.770 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:07.770 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:07.771 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:07.771 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:07.771 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:07.771 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:07.772 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:07.772 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:07.772 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:07.772 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:39:07.772 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:07.773 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:39:07.773 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:39:07.773 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:39:07.773 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:07.774 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:39:07.774 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.164606_332.1 2021-09-17T10:39:12.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:12.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:12.777 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:39:12.832 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:39:12.833 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:39:13.013 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:39:13.013 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:39:13.013 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:39:13.014 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:39:13.014 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:39:13.014 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:39:13.014 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:13.015 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:13.015 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:39:13.015 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:13.015 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:39:13.016 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:13.016 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:13.016 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:39:13.016 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:13.016 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:39:13.017 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:13.017 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:39:13.017 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:13.017 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:13.018 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:13.018 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:13.018 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:39:13.018 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:13.019 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:13.019 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.240117_333.1", 2021-09-17T10:39:13.019 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:13.020 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:13.020 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:13.020 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.240117Z", 2021-09-17T10:39:13.020 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:13.020 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:13.021 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:13.021 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:13.021 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:39:13.021 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:13.022 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:13.022 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:13.022 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:13.022 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:13.023 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:13.023 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:13.023 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:13.023 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:13.024 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:13.024 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:13.024 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:39:13.025 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:13.025 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:13.025 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:13.025 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:13.025 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:13.026 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:13.026 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:13.026 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:13.026 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:13.027 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:13.027 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:39:13.027 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:13.027 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:39:13.028 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:39:13.028 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:39:13.028 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:13.028 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:39:13.030 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.240117_333.1 2021-09-17T10:39:18.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:18.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:18.033 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:39:18.077 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:39:18.077 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:39:18.200 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:39:18.201 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:39:18.201 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:39:18.201 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:39:18.202 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:39:18.202 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:39:18.202 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:18.202 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:18.203 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:39:18.203 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:18.203 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:39:18.203 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:18.203 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:18.204 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:39:18.204 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:18.204 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:39:18.204 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:18.205 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:39:18.205 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:18.205 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:18.205 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:18.206 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:18.206 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:39:18.206 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:18.206 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:18.206 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.157966_331.1", 2021-09-17T10:39:18.207 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:18.207 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:18.207 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:18.207 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.157966Z", 2021-09-17T10:39:18.208 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:18.208 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:18.208 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:18.208 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:18.209 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:39:18.209 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:18.209 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:18.209 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:18.209 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:18.210 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:18.210 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:18.210 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:18.210 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:18.211 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:18.211 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:18.211 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:18.211 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:39:18.212 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:18.212 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:18.212 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:18.212 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:18.212 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:18.213 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:18.213 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:18.213 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:18.213 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:18.214 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:18.214 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:39:18.214 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:18.214 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:39:18.215 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:39:18.215 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:39:18.215 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:18.215 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:39:18.216 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.157966_331.1 2021-09-17T10:39:23.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:23.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:23.219 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:39:23.339 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:39:23.340 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:39:23.731 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:39:23.732 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:39:23.732 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:39:23.732 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:39:23.733 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:39:23.733 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:39:23.733 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:23.733 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:23.734 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:39:23.734 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:23.734 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:39:23.734 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:23.735 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:23.735 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:39:23.735 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:23.735 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:39:23.736 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:23.736 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:39:23.736 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:23.736 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:23.736 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:23.737 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:23.737 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:39:23.737 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:23.737 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:23.738 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:39:23.738 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:23.738 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:23.738 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:39:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:23.740 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:23.740 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:39:23.740 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:23.740 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:23.740 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:23.741 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:23.741 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:23.741 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:23.741 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:23.742 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:23.742 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:23.742 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:23.742 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:23.743 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:39:23.743 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:23.743 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:23.743 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:23.744 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:23.744 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:23.744 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:23.744 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:23.745 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:23.745 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:23.745 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:23.745 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:39:23.745 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:23.746 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:39:23.746 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:39:23.746 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:39:23.746 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:23.747 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:39:23.748 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:39:28.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:28.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:28.750 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:39:28.790 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:39:28.790 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:39:28.911 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:39:28.912 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:39:28.912 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:39:28.912 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:39:28.913 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:39:28.913 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:39:28.913 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:28.913 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:28.914 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:39:28.914 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:28.914 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:39:28.914 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:28.915 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:28.915 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:39:28.915 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:28.915 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:39:28.915 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:28.916 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:39:28.916 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:28.916 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:28.916 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:28.917 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:28.917 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:39:28.917 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:28.917 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:28.917 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:39:28.918 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:28.918 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:28.918 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:28.918 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:39:28.918 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:28.919 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:28.919 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:28.919 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:28.919 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:39:28.920 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:28.920 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:28.920 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:28.920 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:28.920 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:28.921 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:28.921 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:28.921 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:28.921 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:28.921 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:28.922 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:28.922 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:39:28.922 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:28.922 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:28.923 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:28.923 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:28.923 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:28.923 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:28.924 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:28.924 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:28.924 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:28.924 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:28.924 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:39:28.925 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:28.925 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:39:28.925 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:39:28.925 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:39:28.926 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:28.926 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:39:28.927 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:39:33.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:33.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:33.930 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:39:33.972 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:39:33.973 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:39:34.100 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:39:34.100 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:39:34.100 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:39:34.101 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:39:34.101 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:39:34.101 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:39:34.101 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:34.102 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:34.102 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:39:34.102 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:34.102 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:39:34.103 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:34.103 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:34.103 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:39:34.103 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:34.104 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:39:34.104 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:34.104 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:39:34.104 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:34.105 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:34.105 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:34.105 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:34.105 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:39:34.105 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:34.106 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:34.106 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.044111_328.1", 2021-09-17T10:39:34.106 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:34.106 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:34.107 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:34.107 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.044111Z", 2021-09-17T10:39:34.107 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:34.107 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:34.108 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:34.108 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:34.108 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:39:34.108 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:34.108 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:34.109 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:34.109 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:34.109 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:34.109 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:34.110 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:34.110 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:34.110 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:34.110 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:34.111 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:34.111 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:39:34.111 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:34.111 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:34.111 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:34.112 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:34.112 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:34.112 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:34.112 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:34.113 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:34.113 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:34.113 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:34.113 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:39:34.113 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:34.114 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:39:34.114 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:39:34.114 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:39:34.114 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:34.115 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:39:34.116 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.044111_328.1 2021-09-17T10:39:39.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:39.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:39.119 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:39:39.233 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:39:39.234 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:39:39.600 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:39:39.601 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:39:39.601 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:39:39.601 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:39:39.601 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:39:39.602 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:39:39.602 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:39.602 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:39.602 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:39:39.603 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:39.603 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:39:39.603 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:39.603 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:39.604 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:39:39.604 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:39.604 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:39:39.604 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:39.605 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:39:39.605 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:39.605 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:39.605 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:39.605 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:39.606 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:39:39.606 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:39.606 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:39.606 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:39:39.607 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:39.607 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:39.607 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:39.607 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:39:39.608 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:39.608 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:39.608 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:39.608 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:39.608 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:39:39.609 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:39.609 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:39.609 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:39.609 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:39.610 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:39.610 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:39.610 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:39.610 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:39.611 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:39.611 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:39.611 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:39.611 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:39:39.611 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:39.612 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:39.612 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:39.612 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:39.612 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:39.613 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:39.613 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:39.613 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:39.613 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:39.613 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:39.614 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:39:39.614 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:39.614 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:39:39.614 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:39:39.615 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:39:39.615 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:39.615 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:39:39.616 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:39:44.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:44.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:44.619 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:39:44.662 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:39:44.662 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:39:44.787 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:39:44.787 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:39:44.787 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:39:44.788 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:39:44.788 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:39:44.788 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:39:44.788 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:44.789 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:44.789 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:39:44.789 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:44.790 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:39:44.790 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:44.790 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:44.791 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:39:44.791 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:44.791 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:39:44.791 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:44.792 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:39:44.792 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:44.792 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:44.792 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:44.793 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:44.793 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:39:44.793 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:44.793 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:44.794 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.099168_329.1", 2021-09-17T10:39:44.794 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:44.794 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:44.794 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:44.795 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.099168Z", 2021-09-17T10:39:44.795 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:44.795 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:44.795 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:44.795 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:44.796 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:39:44.796 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:44.796 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:44.797 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:44.797 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:44.797 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:44.797 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:44.798 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:44.798 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:44.798 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:44.798 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:44.799 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:44.799 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:39:44.799 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:44.799 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:44.800 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:44.800 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:44.800 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:44.800 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:44.800 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:44.801 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:44.801 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:44.801 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:44.801 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:39:44.801 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:44.802 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:39:44.802 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:39:44.802 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:39:44.802 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:44.802 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:39:44.804 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.099168_329.1 2021-09-17T10:39:49.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:49.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:49.807 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:39:49.865 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:39:49.865 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:39:50.238 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:39:50.239 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:39:50.239 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:39:50.239 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:39:50.239 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:39:50.240 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:39:50.240 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:50.240 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:50.240 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:39:50.240 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:50.241 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:39:50.241 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:50.241 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:50.241 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:39:50.242 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:50.242 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:39:50.242 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:50.242 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:39:50.242 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:50.243 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:50.243 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:50.243 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:50.243 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:39:50.244 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:50.244 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:50.244 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:39:50.244 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:50.244 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:50.245 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:50.245 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:39:50.245 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:50.245 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:50.245 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:50.246 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:50.246 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:39:50.246 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:50.246 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:50.247 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:50.247 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:50.247 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:50.247 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:50.247 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:50.248 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:50.248 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:50.248 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:50.248 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:50.249 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:39:50.249 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:50.249 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:50.249 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:50.249 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:50.250 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:50.250 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:50.250 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:50.250 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:50.251 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:50.251 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:50.251 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:39:50.251 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:50.251 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:39:50.252 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:39:50.252 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:39:50.252 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:50.252 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:39:50.254 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:39:55.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:55.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:39:55.257 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:39:56.345 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:39:56.345 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:39:56.605 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:39:56.606 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:39:56.606 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:39:56.606 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:39:56.606 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:39:56.607 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:39:56.607 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:56.607 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:56.608 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:39:56.608 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:56.609 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:39:56.609 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:56.609 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:56.610 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:39:56.610 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:56.611 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:39:56.611 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:56.612 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:39:56.612 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:56.613 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:56.613 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:56.613 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:56.614 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:39:56.614 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:56.615 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:56.615 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.001301_323.1", 2021-09-17T10:39:56.616 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:56.616 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:56.617 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:56.617 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.001301Z", 2021-09-17T10:39:56.617 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:56.618 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:56.618 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:56.619 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:56.619 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:39:56.619 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:56.619 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:56.620 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:56.620 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:56.620 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:56.621 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:56.621 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:56.621 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:56.621 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:56.622 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:56.622 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:39:56.622 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:39:56.623 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:39:56.623 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:39:56.623 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:39:56.623 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:39:56.624 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:39:56.624 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:39:56.624 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:39:56.624 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:39:56.624 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:56.625 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:56.625 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:39:56.625 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:39:56.625 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:39:56.626 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:39:56.626 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:39:56.626 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:39:56.626 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:39:56.628 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.001301_323.1 2021-09-17T10:40:01.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:01.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:01.631 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:40:01.721 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:40:01.722 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:40:01.997 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:40:01.997 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:40:01.997 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:40:01.998 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:40:01.998 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:40:01.998 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:40:01.998 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:01.999 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:01.999 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:40:01.999 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:01.999 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:40:01.999 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:02.000 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:02.000 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:40:02.000 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:02.000 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:40:02.001 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:02.001 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:40:02.001 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:02.001 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:02.002 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:02.002 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:02.002 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:40:02.002 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:02.002 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:02.003 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.099168_329.1", 2021-09-17T10:40:02.003 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:02.003 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:02.003 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:02.004 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.099168Z", 2021-09-17T10:40:02.004 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:02.004 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:02.004 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:02.005 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:02.005 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:40:02.005 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:02.005 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:02.005 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:02.006 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:02.006 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:02.006 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:02.006 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:02.007 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:02.007 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:02.007 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:02.007 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:02.008 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:40:02.008 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:02.008 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:02.008 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:02.009 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:02.009 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:02.009 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:02.009 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:02.010 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:02.010 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:02.010 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:02.010 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:40:02.011 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:02.011 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:40:02.011 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:40:02.012 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:40:02.012 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:02.012 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:40:02.013 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.099168_329.1 2021-09-17T10:40:07.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:07.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:07.015 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:40:07.059 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:40:07.059 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:40:07.180 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:40:07.181 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:40:07.181 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:40:07.181 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:40:07.181 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:40:07.182 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:40:07.182 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:07.182 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:07.182 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:40:07.183 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:07.183 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:40:07.183 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:07.183 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:07.184 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:40:07.184 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:07.184 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:40:07.184 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:07.184 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:40:07.185 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:07.185 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:07.185 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:07.185 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:07.186 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:40:07.186 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:07.186 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:07.186 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.007158_324.1", 2021-09-17T10:40:07.187 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:07.187 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:07.187 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:07.187 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.007158Z", 2021-09-17T10:40:07.187 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:07.188 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:07.188 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:07.188 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:07.188 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:40:07.189 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:07.189 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:07.189 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:07.189 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:07.189 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:07.190 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:07.190 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:07.190 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:07.190 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:07.191 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:07.191 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:07.191 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:40:07.191 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:07.191 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:07.192 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:07.192 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:07.192 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:07.192 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:07.192 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:07.193 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:07.193 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:07.193 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:07.193 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:40:07.194 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:07.194 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:40:07.194 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:40:07.194 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:40:07.194 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:07.195 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:40:07.196 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.007158_324.1 2021-09-17T10:40:12.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:12.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:12.198 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:40:12.274 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:40:12.274 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:40:12.653 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:40:12.654 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:40:12.654 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:40:12.654 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:40:12.655 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:40:12.655 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:40:12.655 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:12.655 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:12.656 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:40:12.656 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:12.656 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:40:12.656 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:12.656 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:12.657 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:40:12.657 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:12.657 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:40:12.657 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:12.658 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:40:12.658 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:12.658 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:12.658 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:12.659 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:12.659 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:40:12.659 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:12.659 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:12.660 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:40:12.660 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:12.660 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:12.660 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:12.661 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:40:12.661 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:12.661 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:12.661 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:12.661 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:12.662 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:40:12.662 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:12.663 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:12.663 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:12.663 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:12.664 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:12.664 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:12.664 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:12.664 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:12.665 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:12.665 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:12.665 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:12.665 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:40:12.666 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:12.666 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:12.666 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:12.666 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:12.666 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:12.667 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:12.667 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:12.667 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:12.667 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:12.668 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:12.668 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:40:12.668 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:12.668 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:40:12.669 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:40:12.669 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:40:12.669 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:12.669 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:40:12.670 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:40:17.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:17.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:17.673 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:40:17.715 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:40:17.716 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:40:17.807 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:40:17.807 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:40:17.808 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:40:17.808 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:40:17.808 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:40:17.808 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:40:17.809 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:17.809 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:17.809 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:40:17.809 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:17.810 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:40:17.810 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:17.810 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:17.810 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:40:17.811 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:17.811 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:40:17.811 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:17.811 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:40:17.812 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:17.812 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:17.812 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:17.812 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:17.812 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:40:17.813 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:17.813 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:17.813 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:40:17.813 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:17.813 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:17.814 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:17.814 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:40:17.814 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:17.814 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:17.815 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:17.815 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:17.815 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:40:17.815 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:17.816 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:17.816 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:17.816 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:17.816 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:17.816 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:17.817 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:17.817 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:17.817 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:17.817 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:17.818 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:17.818 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:40:17.818 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:17.818 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:17.818 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:17.819 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:17.819 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:17.819 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:17.819 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:17.820 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:17.820 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:17.820 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:17.820 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:40:17.820 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:17.821 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:40:17.821 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:40:17.821 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:40:17.821 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:17.822 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:40:17.823 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.952206_320.1 2021-09-17T10:40:22.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:22.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:22.825 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:40:22.868 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:40:22.869 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:40:22.988 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:40:22.989 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:40:22.989 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:40:22.989 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:40:22.989 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:40:22.990 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:40:22.990 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:22.990 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:22.990 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:40:22.991 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:22.991 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:40:22.991 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:22.991 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:22.991 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:40:22.992 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:22.992 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:40:22.992 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:22.992 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:40:22.993 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:22.993 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:22.993 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:22.993 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:22.993 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:40:22.994 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:22.994 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:22.994 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.249312_334.1", 2021-09-17T10:40:22.994 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:22.995 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:22.995 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:22.995 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.249312Z", 2021-09-17T10:40:22.995 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:22.995 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:22.996 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:22.996 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:22.996 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:40:22.996 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:22.997 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:22.997 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:22.997 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:22.997 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:22.997 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:22.998 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:22.998 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:22.998 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:22.998 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:22.999 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:22.999 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:40:22.999 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:22.999 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:23.000 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:23.000 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:23.000 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:23.000 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:23.000 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:23.001 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:23.001 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:23.001 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:23.001 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:40:23.002 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:23.002 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:40:23.002 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:40:23.002 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:40:23.003 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:23.003 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:40:23.004 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.249312_334.1 2021-09-17T10:40:28.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:28.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:28.007 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:40:28.095 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:40:28.096 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:40:28.556 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:40:28.557 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:40:28.557 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:40:28.557 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:40:28.557 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:40:28.558 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:40:28.558 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:28.558 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:28.559 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:40:28.559 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:28.559 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:40:28.559 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:28.560 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:28.560 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:40:28.560 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:28.560 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:40:28.560 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:28.561 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:40:28.561 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:28.561 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:28.561 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:28.562 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:28.562 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:40:28.562 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:28.562 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:28.562 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.157966_331.1", 2021-09-17T10:40:28.563 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:28.563 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:28.563 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:28.563 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.157966Z", 2021-09-17T10:40:28.564 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:28.564 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:28.564 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:28.564 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:28.564 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:40:28.565 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:28.565 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:28.565 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:28.565 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:28.566 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:28.566 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:28.566 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:28.566 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:28.566 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:28.567 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:28.567 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:28.567 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:40:28.567 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:28.568 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:28.568 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:28.568 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:28.568 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:28.569 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:28.569 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:28.569 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:28.569 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:28.569 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:28.570 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:40:28.570 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:28.570 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:40:28.570 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:40:28.571 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:40:28.571 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:28.571 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:40:28.572 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.157966_331.1 2021-09-17T10:40:33.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:33.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:33.575 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:40:33.631 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:40:33.631 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:40:33.770 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:40:33.770 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:40:33.771 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:40:33.771 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:40:33.771 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:40:33.771 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:40:33.772 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:33.772 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:33.772 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:40:33.772 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:33.772 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:40:33.773 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:33.773 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:33.773 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:40:33.773 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:33.774 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:40:33.774 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:33.774 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:40:33.774 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:33.775 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:33.775 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:33.775 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:33.775 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:40:33.776 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:33.776 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:33.776 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.099168_329.1", 2021-09-17T10:40:33.776 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:33.776 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:33.777 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:33.777 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.099168Z", 2021-09-17T10:40:33.777 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:33.777 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:33.778 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:33.778 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:33.778 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:40:33.778 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:33.779 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:33.779 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:33.779 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:33.779 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:33.780 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:33.780 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:33.780 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:33.780 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:33.781 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:33.781 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:33.781 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:40:33.781 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:33.781 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:33.782 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:33.782 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:33.782 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:33.782 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:33.783 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:33.783 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:33.783 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:33.783 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:33.783 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:40:33.784 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:33.784 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:40:33.784 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:40:33.784 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:40:33.785 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:33.785 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:40:33.786 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.099168_329.1 2021-09-17T10:40:38.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:38.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:38.789 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:40:38.832 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:40:38.833 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:40:38.958 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:40:38.959 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:40:38.959 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:40:38.960 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:40:38.960 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:40:38.960 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:40:38.960 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:38.961 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:38.961 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:40:38.961 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:38.961 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:40:38.962 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:38.962 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:38.962 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:40:38.962 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:38.963 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:40:38.963 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:38.963 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:40:38.963 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:38.964 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:38.964 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:38.965 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:38.965 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:40:38.965 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:38.965 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:38.965 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.001301_323.1", 2021-09-17T10:40:38.966 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:38.966 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:38.966 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:38.966 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.001301Z", 2021-09-17T10:40:38.967 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:38.967 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:38.967 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:38.967 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:38.968 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:40:38.968 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:38.968 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:38.968 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:38.968 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:38.969 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:38.969 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:38.969 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:38.970 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:38.970 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:38.970 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:38.971 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:38.971 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:40:38.971 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:38.971 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:38.972 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:38.972 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:38.972 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:38.972 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:38.973 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:38.973 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:38.973 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:38.973 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:38.974 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:40:38.974 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:38.974 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:40:38.974 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:40:38.975 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:40:38.975 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:38.975 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:40:38.976 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.001301_323.1 2021-09-17T10:40:43.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:43.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:43.980 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:40:44.048 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:40:44.048 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:40:44.498 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:40:44.499 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:40:44.499 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:40:44.499 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:40:44.500 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:40:44.500 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:40:44.500 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:44.500 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:44.501 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:40:44.501 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:44.501 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:40:44.501 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:44.501 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:44.502 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:40:44.502 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:44.502 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:40:44.502 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:44.503 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:40:44.503 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:44.503 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:44.503 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:44.504 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:44.504 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:40:44.504 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:44.504 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:44.504 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:40:44.505 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:44.505 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:44.505 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:44.505 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:40:44.506 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:44.506 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:44.506 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:44.506 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:44.507 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:40:44.507 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:44.507 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:44.507 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:44.508 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:44.508 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:44.508 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:44.508 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:44.508 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:44.509 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:44.509 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:44.509 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:44.509 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:40:44.510 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:44.510 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:44.510 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:44.510 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:44.510 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:44.511 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:44.511 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:44.511 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:44.511 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:44.512 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:44.512 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:40:44.512 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:44.512 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:40:44.513 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:40:44.513 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:40:44.513 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:44.513 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:40:44.515 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:40:49.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:49.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:49.517 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:40:49.643 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:40:49.644 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:40:49.764 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:40:49.765 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:40:49.765 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:40:49.765 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:40:49.766 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:40:49.766 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:40:49.766 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:49.766 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:49.767 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:40:49.767 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:49.767 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:40:49.768 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:49.768 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:49.768 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:40:49.768 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:49.769 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:40:49.769 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:49.769 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:40:49.769 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:49.770 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:49.770 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:49.770 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:49.771 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:40:49.771 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:49.771 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:49.771 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.007158_324.1", 2021-09-17T10:40:49.772 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:49.772 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:49.772 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:49.772 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.007158Z", 2021-09-17T10:40:49.772 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:49.773 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:49.773 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:49.773 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:49.773 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:40:49.774 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:49.774 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:49.774 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:49.774 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:49.775 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:49.775 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:49.775 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:49.775 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:49.776 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:49.776 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:49.776 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:49.776 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:40:49.777 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:49.777 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:49.777 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:49.777 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:49.777 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:49.778 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:49.778 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:49.778 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:49.778 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:49.779 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:49.779 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:40:49.779 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:49.779 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:40:49.780 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:40:49.780 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:40:49.780 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:49.780 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:40:49.781 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.007158_324.1 2021-09-17T10:40:54.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:54.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:40:54.784 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:40:54.869 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:40:54.870 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:40:55.197 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:40:55.197 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:40:55.198 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:40:55.198 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:40:55.199 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:40:55.199 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:40:55.199 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:55.200 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:55.200 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:40:55.200 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:55.200 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:40:55.201 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:55.201 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:55.201 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:40:55.201 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:55.201 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:40:55.202 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:55.202 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:40:55.202 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:55.202 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:55.203 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:55.203 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:55.203 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:40:55.203 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:55.203 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:55.204 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.014731_326.1", 2021-09-17T10:40:55.204 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:55.204 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:55.204 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:55.205 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.014731Z", 2021-09-17T10:40:55.205 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:55.205 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:55.205 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:55.205 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:55.206 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:40:55.206 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:55.206 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:55.206 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:55.207 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:55.207 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:55.207 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:55.207 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:55.208 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:55.208 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:55.208 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:55.208 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:40:55.208 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:40:55.209 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:40:55.209 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:40:55.209 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:40:55.209 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:40:55.210 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:40:55.210 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:40:55.210 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:40:55.210 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:40:55.210 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:55.211 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:55.211 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:40:55.211 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:40:55.211 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:40:55.212 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:40:55.212 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:40:55.212 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:40:55.212 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:40:55.213 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.014731_326.1 2021-09-17T10:41:00.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:00.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:00.215 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:41:00.304 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:41:00.304 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:41:00.389 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:41:00.389 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:41:00.390 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:41:00.390 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:41:00.390 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:41:00.390 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:41:00.390 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:00.391 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:00.391 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:41:00.391 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:00.391 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:41:00.392 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:00.392 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:00.392 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:41:00.392 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:00.392 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:41:00.393 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:00.393 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:41:00.393 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:00.393 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:00.394 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:00.394 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:00.394 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:41:00.394 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:00.395 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:00.395 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:41:00.395 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:00.395 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:00.396 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:00.396 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:41:00.396 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:00.396 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:00.396 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:00.397 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:00.397 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:41:00.397 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:00.397 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:00.398 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:00.398 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:00.398 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:00.398 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:00.398 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:00.399 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:00.399 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:00.399 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:00.399 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:00.400 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:41:00.400 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:00.400 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:00.400 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:00.400 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:00.401 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:00.401 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:00.401 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:00.401 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:00.402 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:00.402 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:00.402 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:41:00.402 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:00.402 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:41:00.403 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:41:00.403 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:41:00.403 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:00.403 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:41:00.405 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:41:05.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:05.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:05.408 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:41:05.451 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:41:05.452 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:41:05.581 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:41:05.582 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:41:05.582 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:41:05.582 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:41:05.582 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:41:05.583 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:41:05.583 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:05.583 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:05.583 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:41:05.584 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:05.584 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:41:05.584 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:05.584 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:05.584 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:41:05.585 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:05.585 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:41:05.585 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:05.585 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:41:05.586 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:05.586 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:05.586 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:05.586 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:05.586 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:41:05.587 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:05.587 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:05.587 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.164606_332.1", 2021-09-17T10:41:05.587 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:05.588 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:05.588 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:05.588 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.164606Z", 2021-09-17T10:41:05.588 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:05.588 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:05.589 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:05.589 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:05.589 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:41:05.590 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:05.590 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:05.590 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:05.591 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:05.591 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:05.591 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:05.591 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:05.592 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:05.592 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:05.592 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:05.592 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:05.592 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:41:05.593 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:05.593 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:05.593 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:05.594 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:05.594 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:05.594 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:05.594 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:05.595 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:05.595 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:05.595 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:05.595 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:41:05.596 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:05.596 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:41:05.596 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:41:05.596 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:41:05.596 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:05.597 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:41:05.598 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.164606_332.1 2021-09-17T10:41:10.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:10.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:10.601 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:41:10.693 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:41:10.694 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:41:11.100 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:41:11.101 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:41:11.101 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:41:11.101 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:41:11.102 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:41:11.102 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:41:11.102 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:11.102 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:11.103 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:41:11.103 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:11.103 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:41:11.103 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:11.103 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:11.104 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:41:11.104 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:11.104 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:41:11.104 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:11.105 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:41:11.105 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:11.105 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:11.105 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:11.106 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:11.106 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:41:11.106 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:11.106 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:11.106 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:41:11.107 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:11.107 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:11.107 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:11.107 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:41:11.108 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:11.108 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:11.108 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:11.109 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:11.109 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:41:11.109 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:11.109 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:11.110 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:11.110 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:11.110 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:11.110 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:11.111 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:11.111 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:11.111 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:11.111 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:11.111 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:11.112 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:41:11.112 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:11.112 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:11.112 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:11.113 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:11.113 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:11.113 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:11.113 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:11.113 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:11.114 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:11.114 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:11.114 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:41:11.114 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:11.115 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:41:11.115 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:41:11.115 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:41:11.115 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:11.116 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:41:11.116 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:41:16.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:16.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:16.119 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:41:16.214 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:41:16.214 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:41:16.339 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:41:16.340 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:41:16.340 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:41:16.340 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:41:16.341 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:41:16.341 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:41:16.341 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:16.341 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:16.342 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:41:16.342 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:16.342 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:41:16.342 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:16.343 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:16.343 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:41:16.343 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:16.343 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:41:16.344 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:16.344 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:41:16.344 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:16.344 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:16.345 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:16.345 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:16.345 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:41:16.345 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:16.346 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:16.346 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.256150_335.1", 2021-09-17T10:41:16.346 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:16.346 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:16.347 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:16.347 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.256150Z", 2021-09-17T10:41:16.347 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:16.347 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:16.348 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:16.348 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:16.348 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:41:16.349 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:16.349 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:16.349 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:16.349 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:16.350 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:16.350 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:16.350 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:16.350 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:16.351 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:16.351 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:16.351 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:16.351 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:41:16.351 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:16.352 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:16.352 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:16.352 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:16.352 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:16.353 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:16.353 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:16.353 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:16.353 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:16.354 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:16.354 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:41:16.354 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:16.354 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:41:16.355 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:41:16.355 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:41:16.355 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:16.355 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:41:16.361 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.256150_335.1 2021-09-17T10:41:21.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:21.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:21.364 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:41:21.572 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:41:21.572 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:41:22.034 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:41:22.034 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:41:22.035 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:41:22.035 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:41:22.035 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:41:22.035 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:41:22.036 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:22.036 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:22.036 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:41:22.036 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:22.037 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:41:22.037 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:22.037 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:22.037 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:41:22.038 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:22.038 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:41:22.038 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:22.038 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:41:22.039 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:22.039 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:22.039 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:22.039 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:22.040 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:41:22.040 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:22.040 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:22.040 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.578535_350.1", 2021-09-17T10:41:22.041 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:22.041 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:22.041 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:22.041 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.578535Z", 2021-09-17T10:41:22.041 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:22.042 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:22.042 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:22.042 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:22.042 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:41:22.043 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:22.043 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:22.043 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:22.043 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:22.044 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:22.044 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:22.044 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:22.044 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:22.044 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:22.045 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:22.045 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:22.045 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:41:22.045 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:22.046 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:22.046 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:22.046 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:22.046 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:22.047 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:22.047 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:22.047 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:22.047 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:22.047 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:22.048 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:41:22.048 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:22.048 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:41:22.048 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:41:22.049 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:41:22.049 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:22.049 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:41:22.051 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.578535_350.1 2021-09-17T10:41:27.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:27.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:27.053 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:41:27.096 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:41:27.096 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:41:27.185 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:41:27.185 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:41:27.186 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:41:27.186 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:41:27.187 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:41:27.187 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:41:27.187 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:27.187 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:27.187 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:41:27.188 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:27.188 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:41:27.188 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:27.189 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:27.189 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:41:27.190 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:27.190 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:41:27.190 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:27.191 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:41:27.191 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:27.192 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:27.192 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:27.193 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:27.193 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:41:27.193 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:27.194 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:27.194 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.038405_327.1", 2021-09-17T10:41:27.195 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:27.195 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:27.196 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:27.196 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.038405Z", 2021-09-17T10:41:27.196 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:27.197 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:27.198 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:27.198 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:27.198 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:41:27.199 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:27.199 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:27.199 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:27.200 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:27.200 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:27.200 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:27.200 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:27.201 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:27.201 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:27.201 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:27.202 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:27.202 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:41:27.203 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:27.203 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:27.204 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:27.204 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:27.204 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:27.204 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:27.204 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:27.205 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:27.205 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:27.205 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:27.206 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:41:27.206 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:27.206 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:41:27.206 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:41:27.207 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:41:27.207 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:27.207 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:41:27.209 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.038405_327.1 2021-09-17T10:41:32.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:32.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:32.211 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:41:32.254 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:41:32.255 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:41:32.373 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:41:32.373 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:41:32.373 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:41:32.373 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:41:32.374 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:41:32.374 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:41:32.374 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:32.374 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:32.375 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:41:32.375 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:32.375 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:41:32.375 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:32.376 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:32.376 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:41:32.376 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:32.376 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:41:32.376 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:32.377 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:41:32.377 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:32.377 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:32.377 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:32.378 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:32.378 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:41:32.378 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:32.378 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:32.378 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.959198_322.1", 2021-09-17T10:41:32.379 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:32.379 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:32.379 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:32.379 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.959198Z", 2021-09-17T10:41:32.380 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:32.380 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:32.380 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:32.380 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:32.381 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:41:32.381 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:32.381 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:32.381 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:32.382 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:32.382 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:32.382 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:32.382 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:32.383 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:32.383 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:32.383 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:32.383 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:32.383 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:41:32.384 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:32.384 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:32.384 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:32.384 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:32.385 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:32.385 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:32.385 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:32.385 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:32.385 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:32.386 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:32.386 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:41:32.386 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:32.386 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:41:32.387 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:41:32.387 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:41:32.387 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:32.387 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:41:32.388 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.959198_322.1 2021-09-17T10:41:37.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:37.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:37.392 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:41:37.450 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:41:37.451 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:41:37.860 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:41:37.861 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:41:37.861 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:41:37.861 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:41:37.862 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:41:37.862 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:41:37.862 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:37.862 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:37.863 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:41:37.863 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:37.863 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:41:37.863 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:37.863 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:37.864 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:41:37.864 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:37.864 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:41:37.864 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:37.864 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:41:37.865 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:37.865 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:37.865 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:37.865 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:37.865 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:41:37.866 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:37.866 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:37.866 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:41:37.866 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:37.867 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:37.867 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:37.867 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:41:37.867 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:37.867 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:37.868 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:37.868 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:37.868 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:41:37.869 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:37.869 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:37.869 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:37.869 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:37.869 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:37.870 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:37.870 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:37.870 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:37.871 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:37.871 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:37.871 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:37.871 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:41:37.872 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:37.872 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:37.872 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:37.872 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:37.872 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:37.873 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:37.873 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:37.873 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:37.873 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:37.874 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:37.874 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:41:37.874 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:37.874 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:41:37.875 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:41:37.875 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:41:37.875 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:37.875 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:41:37.876 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.952206_320.1 2021-09-17T10:41:42.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:42.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:42.879 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:41:43.198 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:41:43.198 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:41:43.318 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:41:43.319 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:41:43.319 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:41:43.319 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:41:43.320 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:41:43.320 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:41:43.320 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:43.320 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:43.320 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:41:43.321 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:43.321 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:41:43.321 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:43.321 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:43.322 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:41:43.322 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:43.322 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:41:43.322 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:43.323 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:41:43.323 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:43.323 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:43.323 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:43.324 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:43.324 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:41:43.324 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:43.324 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:43.324 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:41:43.325 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:43.325 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:43.325 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:43.325 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:41:43.326 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:43.326 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:43.326 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:43.326 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:43.327 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:41:43.327 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:43.327 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:43.327 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:43.328 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:43.328 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:43.328 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:43.328 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:43.329 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:43.329 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:43.329 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:43.329 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:43.330 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:41:43.330 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:43.330 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:43.330 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:43.330 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:43.331 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:43.331 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:43.331 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:43.331 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:43.332 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:43.332 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:43.332 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:41:43.332 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:43.332 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:41:43.333 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:41:43.333 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:41:43.333 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:43.333 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:41:43.335 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:41:48.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:48.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:48.337 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:41:48.498 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:41:48.498 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:41:48.852 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:41:48.852 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:41:48.853 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:41:48.853 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:41:48.853 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:41:48.853 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:41:48.854 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:48.854 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:48.854 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:41:48.855 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:48.855 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:41:48.855 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:48.855 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:48.856 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:41:48.856 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:48.856 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:41:48.856 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:48.856 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:41:48.857 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:48.857 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:48.857 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:48.857 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:48.858 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:41:48.858 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:48.858 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:48.858 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:41:48.859 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:48.859 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:48.859 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:48.859 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:41:48.859 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:48.860 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:48.860 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:48.860 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:48.860 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:41:48.861 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:48.861 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:48.861 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:48.861 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:48.861 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:48.862 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:48.862 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:48.862 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:48.862 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:48.863 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:48.863 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:48.863 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:41:48.863 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:48.864 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:48.864 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:48.864 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:48.864 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:48.865 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:48.865 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:48.865 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:48.865 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:48.866 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:48.866 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:41:48.866 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:48.866 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:41:48.867 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:41:48.867 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:41:48.867 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:48.867 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:41:49.197 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:41:54.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:54.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:54.200 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:41:54.245 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:41:54.245 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:41:54.358 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:41:54.358 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:41:54.358 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:41:54.359 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:41:54.359 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:41:54.359 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:41:54.360 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:54.360 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:54.360 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:41:54.361 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:54.361 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:41:54.362 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:54.362 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:54.362 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:41:54.362 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:54.363 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:41:54.363 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:54.363 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:41:54.363 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:54.364 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:54.364 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:54.364 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:54.364 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:41:54.364 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:54.365 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:54.365 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.959198_322.1", 2021-09-17T10:41:54.365 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:54.366 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:54.366 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:54.366 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.959198Z", 2021-09-17T10:41:54.366 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:54.367 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:54.367 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:54.367 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:54.367 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:41:54.368 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:54.368 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:54.368 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:54.368 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:54.369 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:54.369 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:54.369 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:54.370 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:54.370 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:54.370 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:54.371 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:54.371 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:41:54.371 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:54.371 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:54.372 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:54.372 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:54.372 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:54.373 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:54.373 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:54.373 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:54.373 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:54.374 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:54.374 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:41:54.374 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:54.374 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:41:54.375 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:41:54.375 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:41:54.375 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:54.375 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:41:54.377 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.959198_322.1 2021-09-17T10:41:59.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:59.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:41:59.379 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:41:59.444 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:41:59.444 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:41:59.803 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:41:59.804 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:41:59.804 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:41:59.804 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:41:59.804 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:41:59.805 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:41:59.805 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:59.805 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:59.805 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:41:59.806 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:59.806 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:41:59.806 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:59.806 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:59.807 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:41:59.807 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:59.807 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:41:59.807 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:59.808 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:41:59.808 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:59.808 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:59.808 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:59.809 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:59.809 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:41:59.809 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:59.810 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:59.810 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:41:59.810 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:59.810 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:59.811 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:59.811 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:41:59.811 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:59.811 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:59.812 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:59.812 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:59.812 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:41:59.812 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:59.813 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:59.813 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:59.813 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:59.813 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:59.814 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:59.814 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:59.814 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:59.814 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:59.814 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:59.815 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:41:59.815 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:41:59.815 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:41:59.815 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:41:59.816 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:41:59.816 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:41:59.816 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:41:59.816 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:41:59.817 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:41:59.817 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:41:59.817 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:59.817 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:59.817 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:41:59.818 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:41:59.818 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:41:59.818 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:41:59.818 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:41:59.819 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:41:59.819 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:41:59.819 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:42:04.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:04.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:04.822 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:42:04.867 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:42:04.867 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:42:04.956 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:42:04.957 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:42:04.957 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:42:04.957 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:42:04.957 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:42:04.958 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:42:04.958 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:04.958 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:04.958 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:42:04.959 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:04.959 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:42:04.959 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:04.959 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:04.959 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:42:04.960 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:04.960 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:42:04.960 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:04.960 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:42:04.961 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:04.961 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:04.961 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:04.961 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:04.962 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:42:04.962 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:04.962 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:04.962 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.105095_330.1", 2021-09-17T10:42:04.962 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:04.963 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:04.963 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:04.963 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.105095Z", 2021-09-17T10:42:04.963 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:04.964 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:04.964 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:04.964 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:04.964 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:42:04.965 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:04.965 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:04.965 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:04.965 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:04.966 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:04.966 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:04.966 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:04.966 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:04.966 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:04.967 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:04.967 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:04.967 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:42:04.967 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:04.968 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:04.968 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:04.968 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:04.968 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:04.968 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:04.969 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:04.969 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:04.969 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:04.969 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:04.970 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:42:04.970 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:04.970 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:42:04.970 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:42:04.970 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:42:04.971 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:04.971 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:42:04.972 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.105095_330.1 2021-09-17T10:42:09.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:09.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:09.976 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:42:10.088 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:42:10.089 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:42:10.202 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:42:10.203 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:42:10.203 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:42:10.203 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:42:10.204 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:42:10.204 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:42:10.204 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:10.204 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:10.205 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:42:10.205 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:10.205 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:42:10.205 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:10.205 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:10.206 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:42:10.206 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:10.206 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:42:10.206 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:10.207 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:42:10.207 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:10.207 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:10.207 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:10.208 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:10.208 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:42:10.208 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:10.208 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:10.208 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.001301_323.1", 2021-09-17T10:42:10.209 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:10.209 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:10.209 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:10.209 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.001301Z", 2021-09-17T10:42:10.209 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:10.210 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:10.210 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:10.210 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:10.210 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:42:10.211 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:10.211 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:10.211 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:10.211 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:10.212 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:10.212 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:10.212 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:10.212 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:10.212 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:10.213 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:10.213 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:10.213 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:42:10.213 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:10.214 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:10.214 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:10.214 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:10.214 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:10.214 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:10.215 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:10.215 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:10.215 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:10.215 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:10.216 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:42:10.216 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:10.216 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:42:10.216 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:42:10.216 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:42:10.217 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:10.217 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:42:10.218 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.001301_323.1 2021-09-17T10:42:15.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:15.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:15.221 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:42:15.296 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:42:15.297 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:42:15.648 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:42:15.649 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:42:15.649 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:42:15.649 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:42:15.650 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:42:15.650 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:42:15.650 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:15.650 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:15.651 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:42:15.651 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:15.651 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:42:15.651 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:15.651 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:15.652 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:42:15.652 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:15.652 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:42:15.652 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:15.653 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:42:15.653 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:15.653 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:15.653 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:15.653 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:15.654 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:42:15.654 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:15.654 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:15.654 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:42:15.655 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:15.655 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:15.655 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:15.655 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:42:15.656 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:15.656 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:15.656 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:15.656 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:15.656 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:42:15.657 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:15.657 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:15.657 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:15.657 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:15.658 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:15.658 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:15.658 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:15.658 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:15.658 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:15.659 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:15.659 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:15.659 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:42:15.659 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:15.660 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:15.660 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:15.660 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:15.660 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:15.660 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:15.661 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:15.661 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:15.661 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:15.661 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:15.662 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:42:15.662 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:15.662 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:42:15.662 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:42:15.662 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:42:15.663 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:15.663 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:42:15.664 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:42:20.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:20.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:20.667 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:42:20.740 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:42:20.741 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:42:20.877 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:42:20.877 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:42:20.878 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:42:20.878 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:42:20.878 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:42:20.879 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:42:20.879 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:20.879 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:20.879 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:42:20.880 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:20.880 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:42:20.880 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:20.880 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:20.880 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:42:20.881 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:20.881 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:42:20.881 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:20.881 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:42:20.882 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:20.882 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:20.882 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:20.882 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:20.883 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:42:20.883 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:20.883 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:20.884 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.954369_321.1", 2021-09-17T10:42:20.884 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:20.884 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:20.884 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:20.885 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.954369Z", 2021-09-17T10:42:20.885 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:20.885 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:20.885 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:20.886 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:20.886 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:42:20.886 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:20.886 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:20.887 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:20.887 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:20.887 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:20.887 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:20.887 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:20.888 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:20.888 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:20.888 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:20.888 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:20.889 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:42:20.889 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:20.889 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:20.889 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:20.890 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:20.890 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:20.890 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:20.890 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:20.890 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:20.891 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:20.891 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:20.891 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:42:20.891 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:20.892 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:42:20.892 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:42:20.892 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:42:20.892 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:20.893 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:42:20.894 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.954369_321.1 2021-09-17T10:42:25.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:25.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:25.896 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:42:25.940 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:42:25.941 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:42:26.075 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:42:26.075 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:42:26.076 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:42:26.076 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:42:26.076 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:42:26.077 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:42:26.078 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:26.078 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:26.079 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:42:26.079 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:26.079 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:42:26.080 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:26.080 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:26.080 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:42:26.081 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:26.081 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:42:26.082 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:26.082 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:42:26.083 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:26.083 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:26.084 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:26.084 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:26.084 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:42:26.085 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:26.085 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:26.086 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:42:26.086 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:26.086 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:26.087 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:26.087 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:42:26.088 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:26.088 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:26.088 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:26.089 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:26.089 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:42:26.089 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:26.090 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:26.090 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:26.090 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:26.091 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:26.091 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:26.091 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:26.092 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:26.092 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:26.092 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:26.093 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:26.093 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:42:26.094 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:26.094 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:26.094 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:26.095 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:26.095 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:26.095 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:26.096 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:26.096 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:26.096 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:26.097 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:26.097 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:42:26.098 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:26.098 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:42:26.098 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:42:26.099 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:42:26.099 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:26.099 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:42:26.102 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:42:31.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:31.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:31.105 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:42:31.186 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:42:31.187 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:42:31.566 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:42:31.566 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:42:31.567 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:42:31.567 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:42:31.567 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:42:31.567 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:42:31.568 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:31.568 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:31.569 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:42:31.569 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:31.569 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:42:31.570 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:31.570 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:31.570 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:42:31.570 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:31.570 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:42:31.571 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:31.571 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:42:31.571 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:31.571 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:31.572 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:31.572 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:31.572 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:42:31.572 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:31.573 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:31.573 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:42:31.573 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:31.573 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:31.574 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:31.574 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:42:31.574 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:31.574 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:31.575 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:31.575 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:31.575 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:42:31.575 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:31.576 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:31.576 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:31.576 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:31.576 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:31.576 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:31.577 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:31.577 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:31.577 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:31.577 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:31.577 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:31.578 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:42:31.578 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:31.578 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:31.578 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:31.579 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:31.580 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:31.580 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:31.580 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:31.580 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:31.580 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:31.581 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:31.581 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:42:31.582 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:31.582 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:42:31.582 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:42:31.582 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:42:31.582 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:31.583 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:42:31.584 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.952206_320.1 2021-09-17T10:42:36.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:36.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:36.588 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:42:36.630 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:42:36.630 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:42:36.723 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:42:36.723 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:42:36.724 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:42:36.724 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:42:36.724 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:42:36.724 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:42:36.724 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:36.725 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:36.725 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:42:36.725 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:36.725 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:42:36.726 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:36.726 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:36.726 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:42:36.726 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:36.727 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:42:36.727 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:36.728 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:42:36.728 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:36.728 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:36.728 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:36.729 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:36.729 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:42:36.729 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:36.729 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:36.730 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.157966_331.1", 2021-09-17T10:42:36.730 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:36.730 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:36.730 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:36.731 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.157966Z", 2021-09-17T10:42:36.731 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:36.731 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:36.732 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:36.732 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:36.732 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:42:36.732 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:36.732 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:36.733 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:36.733 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:36.733 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:36.733 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:36.734 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:36.734 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:36.734 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:36.734 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:36.735 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:36.735 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:42:36.735 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:36.735 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:36.736 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:36.736 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:36.736 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:36.736 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:36.737 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:36.737 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:36.737 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:36.738 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:36.738 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:42:36.738 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:36.738 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:42:36.739 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:42:36.739 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:42:36.739 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:36.740 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:42:36.741 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.157966_331.1 2021-09-17T10:42:41.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:41.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:41.743 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:42:41.784 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:42:41.784 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:42:41.908 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:42:41.908 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:42:41.908 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:42:41.909 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:42:41.909 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:42:41.909 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:42:41.910 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:41.910 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:41.910 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:42:41.910 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:41.911 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:42:41.911 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:41.911 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:41.911 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:42:41.912 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:41.912 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:42:41.912 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:41.912 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:42:41.912 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:41.913 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:41.913 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:41.913 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:41.913 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:42:41.914 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:41.914 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:41.914 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.324538_337.1", 2021-09-17T10:42:41.914 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:41.914 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:41.915 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:41.915 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.324538Z", 2021-09-17T10:42:41.915 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:41.915 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:41.916 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:41.916 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:41.916 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:42:41.916 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:41.916 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:41.917 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:41.917 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:41.917 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:41.917 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:41.918 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:41.918 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:41.918 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:41.918 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:41.919 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:41.919 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:42:41.919 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:41.919 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:41.919 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:41.920 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:41.920 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:41.920 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:41.920 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:41.921 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:41.921 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:41.921 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:41.921 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:42:41.921 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:41.922 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:42:41.922 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:42:41.922 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:42:41.922 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:41.923 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:42:41.924 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.324538_337.1 2021-09-17T10:42:46.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:46.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:46.927 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:42:47.019 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:42:47.020 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:42:47.432 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:42:47.433 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:42:47.433 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:42:47.433 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:42:47.434 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:42:47.434 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:42:47.434 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:47.434 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:47.435 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:42:47.435 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:47.435 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:42:47.435 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:47.435 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:47.436 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:42:47.436 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:47.436 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:42:47.437 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:47.437 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:42:47.437 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:47.437 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:47.438 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:47.438 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:47.438 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:42:47.438 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:47.438 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:47.439 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.007158_324.1", 2021-09-17T10:42:47.439 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:47.439 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:47.439 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:47.440 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.007158Z", 2021-09-17T10:42:47.440 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:47.440 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:47.440 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:47.441 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:47.441 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:42:47.441 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:47.441 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:47.442 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:47.442 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:47.442 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:47.442 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:47.443 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:47.443 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:47.443 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:47.443 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:47.444 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:47.444 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:42:47.444 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:47.444 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:47.445 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:47.445 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:47.445 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:47.445 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:47.446 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:47.446 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:47.446 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:47.446 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:47.447 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:42:47.447 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:47.447 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:42:47.447 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:42:47.447 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:42:47.448 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:47.448 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:42:47.449 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.007158_324.1 2021-09-17T10:42:52.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:52.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:52.452 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:42:52.495 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:42:52.495 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:42:52.613 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:42:52.613 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:42:52.614 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:42:52.614 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:42:52.614 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:42:52.614 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:42:52.615 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:52.615 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:52.615 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:42:52.615 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:52.615 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:42:52.616 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:52.616 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:52.616 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:42:52.616 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:52.617 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:42:52.617 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:52.617 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:42:52.617 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:52.617 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:52.618 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:52.618 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:52.618 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:42:52.618 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:52.619 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:52.619 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.256150_335.1", 2021-09-17T10:42:52.619 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:52.619 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:52.620 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:52.620 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.256150Z", 2021-09-17T10:42:52.620 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:52.620 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:52.620 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:52.621 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:52.621 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:42:52.621 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:52.621 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:52.622 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:52.622 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:52.622 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:52.622 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:52.622 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:52.623 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:52.623 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:52.623 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:52.623 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:52.624 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:42:52.624 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:52.624 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:52.624 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:52.625 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:52.625 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:52.625 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:52.625 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:52.625 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:52.626 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:52.626 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:52.626 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:42:52.626 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:52.627 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:42:52.627 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:42:52.627 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:42:52.627 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:52.627 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:42:52.629 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.256150_335.1 2021-09-17T10:42:57.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:57.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:42:57.631 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:42:57.672 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:42:57.673 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:42:57.792 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:42:57.793 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:42:57.793 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:42:57.793 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:42:57.793 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:42:57.794 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:42:57.794 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:57.794 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:57.794 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:42:57.795 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:57.795 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:42:57.795 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:57.795 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:57.796 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:42:57.796 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:57.796 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:42:57.796 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:57.797 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:42:57.797 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:57.797 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:57.797 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:57.797 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:57.798 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:42:57.798 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:57.798 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:57.798 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:42:57.799 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:57.799 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:57.799 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:57.799 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:42:57.800 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:57.800 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:57.800 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:57.800 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:57.801 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:42:57.801 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:57.801 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:57.801 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:57.802 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:57.802 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:57.802 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:57.802 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:57.803 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:57.803 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:57.803 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:57.803 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:42:57.803 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:42:57.804 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:42:57.804 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:42:57.804 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:42:57.804 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:42:57.805 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:42:57.805 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:42:57.805 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:42:57.805 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:42:57.805 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:57.806 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:57.806 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:42:57.806 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:42:57.807 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:42:57.807 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:42:57.807 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:42:57.807 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:42:57.808 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:42:57.809 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:43:02.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:43:02.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:43:02.812 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:43:02.878 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:43:02.879 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:43:03.204 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:43:03.205 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:43:03.205 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:43:03.205 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:43:03.205 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:43:03.206 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:43:03.206 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:43:03.206 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:03.206 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:43:03.207 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:03.207 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:43:03.207 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:43:03.207 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:43:03.207 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:43:03.208 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:43:03.208 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:43:03.208 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:43:03.208 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:43:03.209 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:43:03.209 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:03.209 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:03.209 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:03.209 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:43:03.210 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:43:03.210 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:43:03.210 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873652.952206_320.1", 2021-09-17T10:43:03.210 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:43:03.211 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:43:03.211 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:43:03.212 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:12.952206Z", 2021-09-17T10:43:03.212 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:43:03.212 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:03.213 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:03.213 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:03.213 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:43:03.214 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:43:03.214 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:43:03.214 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:43:03.214 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:43:03.214 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:43:03.215 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:43:03.215 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:43:03.216 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:43:03.216 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:03.216 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:03.216 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:03.216 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:43:03.217 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:43:03.217 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:43:03.217 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:43:03.217 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:43:03.217 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:43:03.218 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:43:03.218 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:43:03.218 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:43:03.218 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:03.219 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:03.219 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:43:03.219 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:03.219 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:43:03.220 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:43:03.220 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:43:03.220 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:03.220 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:43:03.222 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873652.952206_320.1 2021-09-17T10:43:08.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:43:08.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:43:08.226 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:43:08.269 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:43:08.269 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:43:08.386 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:43:08.386 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": { 2021-09-17T10:43:08.386 INFO:teuthology.orchestra.run.smithi018.stdout: "info": { 2021-09-17T10:43:08.387 INFO:teuthology.orchestra.run.smithi018.stdout: "status": "sync", 2021-09-17T10:43:08.387 INFO:teuthology.orchestra.run.smithi018.stdout: "num_shards": 4, 2021-09-17T10:43:08.387 INFO:teuthology.orchestra.run.smithi018.stdout: "period": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:43:08.387 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:43:08.388 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:08.388 INFO:teuthology.orchestra.run.smithi018.stdout: "markers": [ 2021-09-17T10:43:08.388 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:08.388 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 0, 2021-09-17T10:43:08.389 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:43:08.389 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:43:08.389 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873468.462214_91.1", 2021-09-17T10:43:08.389 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:43:08.389 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 1, 2021-09-17T10:43:08.390 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:43:08.390 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:11:08.462214Z", 2021-09-17T10:43:08.390 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:43:08.390 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:08.391 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:08.391 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:08.391 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 1, 2021-09-17T10:43:08.391 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:43:08.392 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:43:08.392 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "1_1631873653.009924_325.1", 2021-09-17T10:43:08.392 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:43:08.392 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:43:08.392 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:43:08.393 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "2021-09-17T10:14:13.009924Z", 2021-09-17T10:43:08.393 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:43:08.393 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:08.393 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:08.394 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:08.394 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 2, 2021-09-17T10:43:08.394 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:43:08.394 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:43:08.395 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:43:08.395 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:43:08.395 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:43:08.395 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:43:08.395 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:43:08.396 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:43:08.396 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:08.396 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:08.396 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:08.397 INFO:teuthology.orchestra.run.smithi018.stdout: "key": 3, 2021-09-17T10:43:08.397 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:43:08.397 INFO:teuthology.orchestra.run.smithi018.stdout: "state": 1, 2021-09-17T10:43:08.397 INFO:teuthology.orchestra.run.smithi018.stdout: "marker": "", 2021-09-17T10:43:08.397 INFO:teuthology.orchestra.run.smithi018.stdout: "next_step_marker": "", 2021-09-17T10:43:08.398 INFO:teuthology.orchestra.run.smithi018.stdout: "total_entries": 0, 2021-09-17T10:43:08.398 INFO:teuthology.orchestra.run.smithi018.stdout: "pos": 0, 2021-09-17T10:43:08.398 INFO:teuthology.orchestra.run.smithi018.stdout: "timestamp": "0.000000", 2021-09-17T10:43:08.398 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:43:08.399 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:08.399 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:08.399 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:43:08.399 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:08.399 INFO:teuthology.orchestra.run.smithi018.stdout: "full_sync": { 2021-09-17T10:43:08.400 INFO:teuthology.orchestra.run.smithi018.stdout: "total": 1, 2021-09-17T10:43:08.400 INFO:teuthology.orchestra.run.smithi018.stdout: "complete": 1 2021-09-17T10:43:08.400 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:08.400 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:43:08.401 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1631874691.729253_479.1 target=1_1631873653.009924_325.1 2021-09-17T10:43:13.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2021-09-17T10:43:13.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:13.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:13.406 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:43:13.445 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:43:13.446 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:43:13.526 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:43:13.526 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "3fa9dfb8-4b98-4254-b8e7-38b17c25c1fb", 2021-09-17T10:43:13.526 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "remove_zg", 2021-09-17T10:43:13.526 INFO:teuthology.orchestra.run.smithi018.stdout: "api_name": "remove_zg", 2021-09-17T10:43:13.527 INFO:teuthology.orchestra.run.smithi018.stdout: "is_master": "false", 2021-09-17T10:43:13.527 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [], 2021-09-17T10:43:13.527 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames": [], 2021-09-17T10:43:13.527 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames_s3website": [], 2021-09-17T10:43:13.528 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zone": "", 2021-09-17T10:43:13.528 INFO:teuthology.orchestra.run.smithi018.stdout: "zones": [], 2021-09-17T10:43:13.528 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_targets": [], 2021-09-17T10:43:13.528 INFO:teuthology.orchestra.run.smithi018.stdout: "default_placement": "", 2021-09-17T10:43:13.529 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:43:13.529 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_policy": { 2021-09-17T10:43:13.529 INFO:teuthology.orchestra.run.smithi018.stdout: "groups": [] 2021-09-17T10:43:13.529 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:13.530 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:43:13.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:13.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:13.532 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:43:13.619 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:43:13.620 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:43:13.829 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:43:13.829 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "effcdd8b-09d5-4f06-913b-9c2c0b7a1bd1", 2021-09-17T10:43:13.830 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "remove", 2021-09-17T10:43:13.830 INFO:teuthology.orchestra.run.smithi018.stdout: "domain_root": "remove.rgw.meta:root", 2021-09-17T10:43:13.830 INFO:teuthology.orchestra.run.smithi018.stdout: "control_pool": "remove.rgw.control", 2021-09-17T10:43:13.830 INFO:teuthology.orchestra.run.smithi018.stdout: "gc_pool": "remove.rgw.log:gc", 2021-09-17T10:43:13.831 INFO:teuthology.orchestra.run.smithi018.stdout: "lc_pool": "remove.rgw.log:lc", 2021-09-17T10:43:13.831 INFO:teuthology.orchestra.run.smithi018.stdout: "log_pool": "remove.rgw.log", 2021-09-17T10:43:13.831 INFO:teuthology.orchestra.run.smithi018.stdout: "intent_log_pool": "remove.rgw.log:intent", 2021-09-17T10:43:13.831 INFO:teuthology.orchestra.run.smithi018.stdout: "usage_log_pool": "remove.rgw.log:usage", 2021-09-17T10:43:13.832 INFO:teuthology.orchestra.run.smithi018.stdout: "roles_pool": "remove.rgw.meta:roles", 2021-09-17T10:43:13.832 INFO:teuthology.orchestra.run.smithi018.stdout: "reshard_pool": "remove.rgw.log:reshard", 2021-09-17T10:43:13.832 INFO:teuthology.orchestra.run.smithi018.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2021-09-17T10:43:13.832 INFO:teuthology.orchestra.run.smithi018.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2021-09-17T10:43:13.833 INFO:teuthology.orchestra.run.smithi018.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2021-09-17T10:43:13.833 INFO:teuthology.orchestra.run.smithi018.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2021-09-17T10:43:13.833 INFO:teuthology.orchestra.run.smithi018.stdout: "otp_pool": "remove.rgw.otp", 2021-09-17T10:43:13.834 INFO:teuthology.orchestra.run.smithi018.stdout: "system_key": { 2021-09-17T10:43:13.834 INFO:teuthology.orchestra.run.smithi018.stdout: "access_key": "", 2021-09-17T10:43:13.834 INFO:teuthology.orchestra.run.smithi018.stdout: "secret_key": "" 2021-09-17T10:43:13.834 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:13.834 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_pools": [ 2021-09-17T10:43:13.835 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:13.835 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "default-placement", 2021-09-17T10:43:13.835 INFO:teuthology.orchestra.run.smithi018.stdout: "val": { 2021-09-17T10:43:13.835 INFO:teuthology.orchestra.run.smithi018.stdout: "index_pool": "remove.rgw.buckets.index", 2021-09-17T10:43:13.836 INFO:teuthology.orchestra.run.smithi018.stdout: "storage_classes": { 2021-09-17T10:43:13.836 INFO:teuthology.orchestra.run.smithi018.stdout: "STANDARD": { 2021-09-17T10:43:13.836 INFO:teuthology.orchestra.run.smithi018.stdout: "data_pool": "remove.rgw.buckets.data" 2021-09-17T10:43:13.836 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:13.837 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:13.837 INFO:teuthology.orchestra.run.smithi018.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2021-09-17T10:43:13.837 INFO:teuthology.orchestra.run.smithi018.stdout: "index_type": 0 2021-09-17T10:43:13.837 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:13.837 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:13.838 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:13.838 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:43:13.838 INFO:teuthology.orchestra.run.smithi018.stdout: "notif_pool": "remove.rgw.log:notif" 2021-09-17T10:43:13.838 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:43:13.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:13.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:13.841 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:43:13.939 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:43:13.940 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:43:14.385 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:43:14.386 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:43:14.386 INFO:teuthology.orchestra.run.smithi018.stdout: "epoch": 6, 2021-09-17T10:43:14.386 INFO:teuthology.orchestra.run.smithi018.stdout: "predecessor_uuid": "20a9da20-85d6-489f-bcfe-ebc80031614a", 2021-09-17T10:43:14.386 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": [], 2021-09-17T10:43:14.387 INFO:teuthology.orchestra.run.smithi018.stdout: "period_map": { 2021-09-17T10:43:14.387 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:43:14.387 INFO:teuthology.orchestra.run.smithi018.stdout: "zonegroups": [ 2021-09-17T10:43:14.388 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.388 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "3fa9dfb8-4b98-4254-b8e7-38b17c25c1fb", 2021-09-17T10:43:14.388 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "remove_zg", 2021-09-17T10:43:14.388 INFO:teuthology.orchestra.run.smithi018.stdout: "api_name": "remove_zg", 2021-09-17T10:43:14.388 INFO:teuthology.orchestra.run.smithi018.stdout: "is_master": "false", 2021-09-17T10:43:14.389 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [], 2021-09-17T10:43:14.389 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames": [], 2021-09-17T10:43:14.389 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames_s3website": [], 2021-09-17T10:43:14.389 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zone": "effcdd8b-09d5-4f06-913b-9c2c0b7a1bd1", 2021-09-17T10:43:14.389 INFO:teuthology.orchestra.run.smithi018.stdout: "zones": [ 2021-09-17T10:43:14.390 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.390 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "effcdd8b-09d5-4f06-913b-9c2c0b7a1bd1", 2021-09-17T10:43:14.390 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "remove", 2021-09-17T10:43:14.391 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [], 2021-09-17T10:43:14.391 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:43:14.391 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "false", 2021-09-17T10:43:14.391 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:43:14.392 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:43:14.392 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "", 2021-09-17T10:43:14.392 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "true", 2021-09-17T10:43:14.392 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [], 2021-09-17T10:43:14.392 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:43:14.393 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:14.393 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:14.393 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_targets": [ 2021-09-17T10:43:14.393 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.394 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "default-placement", 2021-09-17T10:43:14.394 INFO:teuthology.orchestra.run.smithi018.stdout: "tags": [], 2021-09-17T10:43:14.394 INFO:teuthology.orchestra.run.smithi018.stdout: "storage_classes": [ 2021-09-17T10:43:14.394 INFO:teuthology.orchestra.run.smithi018.stdout: "STANDARD" 2021-09-17T10:43:14.394 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:43:14.395 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:14.395 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:14.395 INFO:teuthology.orchestra.run.smithi018.stdout: "default_placement": "default-placement", 2021-09-17T10:43:14.395 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:43:14.396 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_policy": { 2021-09-17T10:43:14.396 INFO:teuthology.orchestra.run.smithi018.stdout: "groups": [] 2021-09-17T10:43:14.396 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:14.396 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:14.396 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.397 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "test-zonegroup", 2021-09-17T10:43:14.397 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zonegroup", 2021-09-17T10:43:14.397 INFO:teuthology.orchestra.run.smithi018.stdout: "api_name": "", 2021-09-17T10:43:14.397 INFO:teuthology.orchestra.run.smithi018.stdout: "is_master": "true", 2021-09-17T10:43:14.398 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:43:14.398 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:43:14.398 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:14.398 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames": [], 2021-09-17T10:43:14.398 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames_s3website": [], 2021-09-17T10:43:14.399 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:43:14.401 INFO:teuthology.orchestra.run.smithi018.stdout: "zones": [ 2021-09-17T10:43:14.401 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.401 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:43:14.402 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone2", 2021-09-17T10:43:14.402 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:43:14.402 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi050.front.sepia.ceph.com:8000" 2021-09-17T10:43:14.402 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:14.403 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:43:14.403 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "true", 2021-09-17T10:43:14.403 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:43:14.403 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:43:14.404 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "", 2021-09-17T10:43:14.404 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "false", 2021-09-17T10:43:14.404 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [], 2021-09-17T10:43:14.404 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:43:14.404 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:14.405 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.405 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "74241c6a-1297-4f4a-8bbd-b65c661636d9", 2021-09-17T10:43:14.405 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone4", 2021-09-17T10:43:14.405 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:43:14.406 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi050.front.sepia.ceph.com:8001" 2021-09-17T10:43:14.406 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:14.406 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:43:14.406 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "true", 2021-09-17T10:43:14.406 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:43:14.407 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:43:14.407 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "pubsub", 2021-09-17T10:43:14.407 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "false", 2021-09-17T10:43:14.407 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [ 2021-09-17T10:43:14.408 INFO:teuthology.orchestra.run.smithi018.stdout: "test-zone1" 2021-09-17T10:43:14.408 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:14.408 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:43:14.408 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:14.409 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.409 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:43:14.409 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone1", 2021-09-17T10:43:14.409 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:43:14.409 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:43:14.410 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:14.410 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:43:14.410 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "true", 2021-09-17T10:43:14.410 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:43:14.411 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:43:14.411 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "", 2021-09-17T10:43:14.411 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "true", 2021-09-17T10:43:14.412 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [], 2021-09-17T10:43:14.412 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:43:14.412 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:14.412 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.413 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "d9facd38-bb4e-4ea2-9817-2655cc5c6379", 2021-09-17T10:43:14.413 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone3", 2021-09-17T10:43:14.413 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:43:14.413 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi018.front.sepia.ceph.com:8001" 2021-09-17T10:43:14.414 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:14.414 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:43:14.414 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "true", 2021-09-17T10:43:14.414 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:43:14.414 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:43:14.415 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "", 2021-09-17T10:43:14.415 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "true", 2021-09-17T10:43:14.415 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [], 2021-09-17T10:43:14.415 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:43:14.416 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:14.416 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:14.416 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_targets": [ 2021-09-17T10:43:14.416 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.416 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "default-placement", 2021-09-17T10:43:14.417 INFO:teuthology.orchestra.run.smithi018.stdout: "tags": [], 2021-09-17T10:43:14.417 INFO:teuthology.orchestra.run.smithi018.stdout: "storage_classes": [ 2021-09-17T10:43:14.417 INFO:teuthology.orchestra.run.smithi018.stdout: "STANDARD" 2021-09-17T10:43:14.417 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:43:14.418 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:14.418 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:14.418 INFO:teuthology.orchestra.run.smithi018.stdout: "default_placement": "default-placement", 2021-09-17T10:43:14.418 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:43:14.419 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_policy": { 2021-09-17T10:43:14.419 INFO:teuthology.orchestra.run.smithi018.stdout: "groups": [] 2021-09-17T10:43:14.419 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:14.419 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:14.420 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:14.420 INFO:teuthology.orchestra.run.smithi018.stdout: "short_zone_ids": [ 2021-09-17T10:43:14.420 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.420 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:43:14.420 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 1815852058 2021-09-17T10:43:14.421 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:14.421 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.421 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "74241c6a-1297-4f4a-8bbd-b65c661636d9", 2021-09-17T10:43:14.421 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 1495243106 2021-09-17T10:43:14.422 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:14.422 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.422 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:43:14.422 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 664423460 2021-09-17T10:43:14.422 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:14.423 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.423 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "d9facd38-bb4e-4ea2-9817-2655cc5c6379", 2021-09-17T10:43:14.423 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 2096482420 2021-09-17T10:43:14.423 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:14.424 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:14.424 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "effcdd8b-09d5-4f06-913b-9c2c0b7a1bd1", 2021-09-17T10:43:14.424 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 88156623 2021-09-17T10:43:14.424 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:14.425 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:43:14.425 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:14.425 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T10:43:14.425 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:43:14.425 INFO:teuthology.orchestra.run.smithi018.stdout: "period_config": { 2021-09-17T10:43:14.426 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_quota": { 2021-09-17T10:43:14.426 INFO:teuthology.orchestra.run.smithi018.stdout: "enabled": false, 2021-09-17T10:43:14.426 INFO:teuthology.orchestra.run.smithi018.stdout: "check_on_raw": false, 2021-09-17T10:43:14.426 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size": -1, 2021-09-17T10:43:14.427 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size_kb": 0, 2021-09-17T10:43:14.427 INFO:teuthology.orchestra.run.smithi018.stdout: "max_objects": -1 2021-09-17T10:43:14.427 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:14.427 INFO:teuthology.orchestra.run.smithi018.stdout: "user_quota": { 2021-09-17T10:43:14.428 INFO:teuthology.orchestra.run.smithi018.stdout: "enabled": false, 2021-09-17T10:43:14.428 INFO:teuthology.orchestra.run.smithi018.stdout: "check_on_raw": false, 2021-09-17T10:43:14.428 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size": -1, 2021-09-17T10:43:14.428 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size_kb": 0, 2021-09-17T10:43:14.428 INFO:teuthology.orchestra.run.smithi018.stdout: "max_objects": -1 2021-09-17T10:43:14.429 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:14.429 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:14.429 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:43:14.429 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_name": "test-realm", 2021-09-17T10:43:14.429 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:43:14.430 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:43:14.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:14.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:14.432 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:43:14.468 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:43:14.468 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:43:14.540 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:43:14.558+0000 7f730eb1db00 0 zone id effcdd8b-09d5-4f06-913b-9c2c0b7a1bd1 is not a part of zonegroup test-zonegroup 2021-09-17T10:43:14.541 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:43:14.562+0000 7f730eb1db00 0 zone id effcdd8b-09d5-4f06-913b-9c2c0b7a1bd1 is not a part of zonegroup default 2021-09-17T10:43:14.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:14.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:14.554 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:43:14.592 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:43:14.592 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:43:14.649 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:43:14.670+0000 7f08ad11ab00 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-09-17T10:43:14.655 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-09-17T10:43:14.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:14.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:14.657 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:43:14.695 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:43:14.695 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:43:14.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:14.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-17T10:43:14.769 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-17T10:43:14.859 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:43:14.859 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:43:14.997 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2021-09-17T10:43:14.998 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:43:14.998 INFO:teuthology.orchestra.run.smithi018.stdout: "epoch": 7, 2021-09-17T10:43:14.998 INFO:teuthology.orchestra.run.smithi018.stdout: "predecessor_uuid": "20a9da20-85d6-489f-bcfe-ebc80031614a", 2021-09-17T10:43:14.999 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_status": [], 2021-09-17T10:43:14.999 INFO:teuthology.orchestra.run.smithi018.stdout: "period_map": { 2021-09-17T10:43:14.999 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "3d497f82-4d9a-4e2e-97c3-3bd17c5b3aac", 2021-09-17T10:43:14.999 INFO:teuthology.orchestra.run.smithi018.stdout: "zonegroups": [ 2021-09-17T10:43:15.000 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:15.000 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "test-zonegroup", 2021-09-17T10:43:15.000 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zonegroup", 2021-09-17T10:43:15.000 INFO:teuthology.orchestra.run.smithi018.stdout: "api_name": "", 2021-09-17T10:43:15.001 INFO:teuthology.orchestra.run.smithi018.stdout: "is_master": "true", 2021-09-17T10:43:15.001 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:43:15.001 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:43:15.001 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:15.002 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames": [], 2021-09-17T10:43:15.002 INFO:teuthology.orchestra.run.smithi018.stdout: "hostnames_s3website": [], 2021-09-17T10:43:15.002 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:43:15.002 INFO:teuthology.orchestra.run.smithi018.stdout: "zones": [ 2021-09-17T10:43:15.002 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:15.003 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:43:15.003 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone2", 2021-09-17T10:43:15.003 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:43:15.003 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi050.front.sepia.ceph.com:8000" 2021-09-17T10:43:15.004 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:15.004 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:43:15.004 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "true", 2021-09-17T10:43:15.004 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:43:15.004 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:43:15.005 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "", 2021-09-17T10:43:15.005 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "false", 2021-09-17T10:43:15.005 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [], 2021-09-17T10:43:15.005 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:43:15.006 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:15.006 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:15.006 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "74241c6a-1297-4f4a-8bbd-b65c661636d9", 2021-09-17T10:43:15.006 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone4", 2021-09-17T10:43:15.006 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:43:15.007 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi050.front.sepia.ceph.com:8001" 2021-09-17T10:43:15.007 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:15.007 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:43:15.007 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "true", 2021-09-17T10:43:15.007 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:43:15.008 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:43:15.008 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "pubsub", 2021-09-17T10:43:15.008 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "false", 2021-09-17T10:43:15.008 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [ 2021-09-17T10:43:15.009 INFO:teuthology.orchestra.run.smithi018.stdout: "test-zone1" 2021-09-17T10:43:15.009 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:15.009 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:43:15.009 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:15.010 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:15.010 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:43:15.010 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone1", 2021-09-17T10:43:15.010 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:43:15.011 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi018.front.sepia.ceph.com:8000" 2021-09-17T10:43:15.011 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:15.011 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:43:15.011 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "true", 2021-09-17T10:43:15.012 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:43:15.012 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:43:15.012 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "", 2021-09-17T10:43:15.012 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "true", 2021-09-17T10:43:15.012 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [], 2021-09-17T10:43:15.013 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:43:15.013 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:15.013 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:15.013 INFO:teuthology.orchestra.run.smithi018.stdout: "id": "d9facd38-bb4e-4ea2-9817-2655cc5c6379", 2021-09-17T10:43:15.014 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "test-zone3", 2021-09-17T10:43:15.014 INFO:teuthology.orchestra.run.smithi018.stdout: "endpoints": [ 2021-09-17T10:43:15.014 INFO:teuthology.orchestra.run.smithi018.stdout: "http://smithi018.front.sepia.ceph.com:8001" 2021-09-17T10:43:15.014 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:15.015 INFO:teuthology.orchestra.run.smithi018.stdout: "log_meta": "false", 2021-09-17T10:43:15.015 INFO:teuthology.orchestra.run.smithi018.stdout: "log_data": "true", 2021-09-17T10:43:15.015 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_index_max_shards": 11, 2021-09-17T10:43:15.015 INFO:teuthology.orchestra.run.smithi018.stdout: "read_only": "false", 2021-09-17T10:43:15.016 INFO:teuthology.orchestra.run.smithi018.stdout: "tier_type": "", 2021-09-17T10:43:15.016 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from_all": "true", 2021-09-17T10:43:15.016 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_from": [], 2021-09-17T10:43:15.016 INFO:teuthology.orchestra.run.smithi018.stdout: "redirect_zone": "" 2021-09-17T10:43:15.016 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:15.017 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:15.017 INFO:teuthology.orchestra.run.smithi018.stdout: "placement_targets": [ 2021-09-17T10:43:15.017 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:15.017 INFO:teuthology.orchestra.run.smithi018.stdout: "name": "default-placement", 2021-09-17T10:43:15.018 INFO:teuthology.orchestra.run.smithi018.stdout: "tags": [], 2021-09-17T10:43:15.018 INFO:teuthology.orchestra.run.smithi018.stdout: "storage_classes": [ 2021-09-17T10:43:15.018 INFO:teuthology.orchestra.run.smithi018.stdout: "STANDARD" 2021-09-17T10:43:15.018 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:43:15.019 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:15.019 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:15.019 INFO:teuthology.orchestra.run.smithi018.stdout: "default_placement": "default-placement", 2021-09-17T10:43:15.019 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:43:15.020 INFO:teuthology.orchestra.run.smithi018.stdout: "sync_policy": { 2021-09-17T10:43:15.020 INFO:teuthology.orchestra.run.smithi018.stdout: "groups": [] 2021-09-17T10:43:15.020 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:15.020 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:15.020 INFO:teuthology.orchestra.run.smithi018.stdout: ], 2021-09-17T10:43:15.021 INFO:teuthology.orchestra.run.smithi018.stdout: "short_zone_ids": [ 2021-09-17T10:43:15.021 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:15.022 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "304080b6-3b8b-4555-8935-e35e25942a93", 2021-09-17T10:43:15.022 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 1815852058 2021-09-17T10:43:15.023 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:15.023 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:15.023 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "74241c6a-1297-4f4a-8bbd-b65c661636d9", 2021-09-17T10:43:15.023 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 1495243106 2021-09-17T10:43:15.023 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:15.024 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:15.024 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:43:15.024 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 664423460 2021-09-17T10:43:15.024 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:15.025 INFO:teuthology.orchestra.run.smithi018.stdout: { 2021-09-17T10:43:15.025 INFO:teuthology.orchestra.run.smithi018.stdout: "key": "d9facd38-bb4e-4ea2-9817-2655cc5c6379", 2021-09-17T10:43:15.025 INFO:teuthology.orchestra.run.smithi018.stdout: "val": 2096482420 2021-09-17T10:43:15.025 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:15.025 INFO:teuthology.orchestra.run.smithi018.stdout: ] 2021-09-17T10:43:15.026 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:15.026 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zonegroup": "test-zonegroup", 2021-09-17T10:43:15.026 INFO:teuthology.orchestra.run.smithi018.stdout: "master_zone": "9854c5a3-6cc0-45a7-b530-3af2b1c733d5", 2021-09-17T10:43:15.026 INFO:teuthology.orchestra.run.smithi018.stdout: "period_config": { 2021-09-17T10:43:15.027 INFO:teuthology.orchestra.run.smithi018.stdout: "bucket_quota": { 2021-09-17T10:43:15.027 INFO:teuthology.orchestra.run.smithi018.stdout: "enabled": false, 2021-09-17T10:43:15.027 INFO:teuthology.orchestra.run.smithi018.stdout: "check_on_raw": false, 2021-09-17T10:43:15.027 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size": -1, 2021-09-17T10:43:15.028 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size_kb": 0, 2021-09-17T10:43:15.028 INFO:teuthology.orchestra.run.smithi018.stdout: "max_objects": -1 2021-09-17T10:43:15.028 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:15.028 INFO:teuthology.orchestra.run.smithi018.stdout: "user_quota": { 2021-09-17T10:43:15.028 INFO:teuthology.orchestra.run.smithi018.stdout: "enabled": false, 2021-09-17T10:43:15.029 INFO:teuthology.orchestra.run.smithi018.stdout: "check_on_raw": false, 2021-09-17T10:43:15.029 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size": -1, 2021-09-17T10:43:15.029 INFO:teuthology.orchestra.run.smithi018.stdout: "max_size_kb": 0, 2021-09-17T10:43:15.029 INFO:teuthology.orchestra.run.smithi018.stdout: "max_objects": -1 2021-09-17T10:43:15.029 INFO:teuthology.orchestra.run.smithi018.stdout: } 2021-09-17T10:43:15.030 INFO:teuthology.orchestra.run.smithi018.stdout: }, 2021-09-17T10:43:15.030 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_id": "3c1ea6e8-af17-477a-9c20-483e62be612c", 2021-09-17T10:43:15.030 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_name": "test-realm", 2021-09-17T10:43:15.030 INFO:teuthology.orchestra.run.smithi018.stdout: "realm_epoch": 2 2021-09-17T10:43:15.031 INFO:teuthology.orchestra.run.smithi018.stdout:} 2021-09-17T10:43:15.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-09-17T10:43:15.032 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-46 2021-09-17T10:43:15.462 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gopfzo-47 2021-09-17T10:43:15.524 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gopfzo-48 2021-09-17T10:43:15.578 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-49 2021-09-17T10:43:15.580 INFO:tasks.rgw.c1.client.0.smithi018.stdout:*** Caught signal (Segmentation fault) ** 2021-09-17T10:43:15.580 INFO:tasks.rgw.c1.client.0.smithi018.stdout: in thread 7f8171eeb700 thread_name:radosgw 2021-09-17T10:43:15.600 INFO:tasks.rgw.c1.client.0.smithi018.stdout: ceph version 17.0.0-7741-gf7edf77e (f7edf77e75bbfd0d8d41a561025fbfc76479a1a3) quincy (dev) 2021-09-17T10:43:15.600 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f831e5083c0] 2021-09-17T10:43:15.600 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x840) [0x7f831eaad310] 2021-09-17T10:43:15.601 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x209f) [0x7f831eacbc1f] 2021-09-17T10:43:15.601 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 4: /lib/libradosgw.so.2(+0x4d2b9d) [0x7f831e9e8b9d] 2021-09-17T10:43:15.601 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 5: /lib/libradosgw.so.2(+0x4d3f81) [0x7f831e9e9f81] 2021-09-17T10:43:15.601 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 6: /lib/libradosgw.so.2(+0x4d40fc) [0x7f831e9ea0fc] 2021-09-17T10:43:15.602 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 7: make_fcontext() 2021-09-17T10:43:15.609 INFO:tasks.rgw.c1.client.0.smithi018.stdout:2021-09-17T10:43:15.618+0000 7f8171eeb700 -1 *** Caught signal (Segmentation fault) ** 2021-09-17T10:43:15.609 INFO:tasks.rgw.c1.client.0.smithi018.stdout: in thread 7f8171eeb700 thread_name:radosgw 2021-09-17T10:43:15.609 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 2021-09-17T10:43:15.609 INFO:tasks.rgw.c1.client.0.smithi018.stdout: ceph version 17.0.0-7741-gf7edf77e (f7edf77e75bbfd0d8d41a561025fbfc76479a1a3) quincy (dev) 2021-09-17T10:43:15.610 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f831e5083c0] 2021-09-17T10:43:15.610 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x840) [0x7f831eaad310] 2021-09-17T10:43:15.610 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x209f) [0x7f831eacbc1f] 2021-09-17T10:43:15.610 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 4: /lib/libradosgw.so.2(+0x4d2b9d) [0x7f831e9e8b9d] 2021-09-17T10:43:15.611 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 5: /lib/libradosgw.so.2(+0x4d3f81) [0x7f831e9e9f81] 2021-09-17T10:43:15.611 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 6: /lib/libradosgw.so.2(+0x4d40fc) [0x7f831e9ea0fc] 2021-09-17T10:43:15.611 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 7: make_fcontext() 2021-09-17T10:43:15.611 INFO:tasks.rgw.c1.client.0.smithi018.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-17T10:43:15.612 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 2021-09-17T10:43:15.619 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 0> 2021-09-17T10:43:15.618+0000 7f8171eeb700 -1 *** Caught signal (Segmentation fault) ** 2021-09-17T10:43:15.619 INFO:tasks.rgw.c1.client.0.smithi018.stdout: in thread 7f8171eeb700 thread_name:radosgw 2021-09-17T10:43:15.619 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 2021-09-17T10:43:15.620 INFO:tasks.rgw.c1.client.0.smithi018.stdout: ceph version 17.0.0-7741-gf7edf77e (f7edf77e75bbfd0d8d41a561025fbfc76479a1a3) quincy (dev) 2021-09-17T10:43:15.620 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f831e5083c0] 2021-09-17T10:43:15.620 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x840) [0x7f831eaad310] 2021-09-17T10:43:15.620 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x209f) [0x7f831eacbc1f] 2021-09-17T10:43:15.620 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 4: /lib/libradosgw.so.2(+0x4d2b9d) [0x7f831e9e8b9d] 2021-09-17T10:43:15.621 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 5: /lib/libradosgw.so.2(+0x4d3f81) [0x7f831e9e9f81] 2021-09-17T10:43:15.621 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 6: /lib/libradosgw.so.2(+0x4d40fc) [0x7f831e9ea0fc] 2021-09-17T10:43:15.621 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 7: make_fcontext() 2021-09-17T10:43:15.621 INFO:tasks.rgw.c1.client.0.smithi018.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-17T10:43:15.622 INFO:tasks.rgw.c1.client.0.smithi018.stdout: 2021-09-17T10:43:15.901 INFO:tasks.rgw.c1.client.0.smithi018.stderr:daemon-helper: command crashed with signal 11 2021-09-17T10:43:16.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-09-17T10:43:22.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2021-09-17T10:43:27.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~11s 2021-09-17T10:43:33.377 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~17s 2021-09-17T10:43:38.982 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~22s 2021-09-17T10:43:44.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~28s 2021-09-17T10:43:50.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~34s 2021-09-17T10:43:55.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~39s 2021-09-17T10:44:01.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~45s 2021-09-17T10:44:07.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~50s 2021-09-17T10:44:12.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~56s 2021-09-17T10:44:18.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~62s 2021-09-17T10:44:23.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~67s 2021-09-17T10:44:29.430 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~73s 2021-09-17T10:44:35.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~78s 2021-09-17T10:44:35.092 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... rgw_multi.tests.test_set_bucket_policy ... ERROR 2021-09-17T10:44:35.093 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-50 2021-09-17T10:44:40.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~84s 2021-09-17T10:44:46.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~90s 2021-09-17T10:44:51.851 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~95s 2021-09-17T10:44:57.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~101s 2021-09-17T10:45:03.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~107s 2021-09-17T10:45:08.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~112s 2021-09-17T10:45:14.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~118s 2021-09-17T10:45:19.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~123s 2021-09-17T10:45:25.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~129s 2021-09-17T10:45:31.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~135s 2021-09-17T10:45:36.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~140s 2021-09-17T10:45:42.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~146s 2021-09-17T10:45:46.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ERROR 2021-09-17T10:45:46.551 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-51 2021-09-17T10:45:47.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~151s 2021-09-17T10:45:53.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~157s 2021-09-17T10:45:59.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~163s 2021-09-17T10:46:04.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~168s 2021-09-17T10:46:10.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~174s 2021-09-17T10:46:15.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~179s 2021-09-17T10:46:21.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~185s 2021-09-17T10:46:27.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~191s 2021-09-17T10:46:32.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~196s 2021-09-17T10:46:38.353 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~202s 2021-09-17T10:46:43.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~207s 2021-09-17T10:46:49.563 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~213s 2021-09-17T10:46:55.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~219s 2021-09-17T10:46:57.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ERROR 2021-09-17T10:46:57.279 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-52 2021-09-17T10:47:00.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~224s 2021-09-17T10:47:06.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~230s 2021-09-17T10:47:11.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~235s 2021-09-17T10:47:17.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~241s 2021-09-17T10:47:23.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~247s 2021-09-17T10:47:28.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~252s 2021-09-17T10:47:34.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~258s 2021-09-17T10:47:40.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~263s 2021-09-17T10:47:45.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~269s 2021-09-17T10:47:51.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~275s 2021-09-17T10:47:56.818 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~280s 2021-09-17T10:48:02.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~286s 2021-09-17T10:48:08.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~291s 2021-09-17T10:48:13.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~297s 2021-09-17T10:48:19.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~303s 2021-09-17T10:48:19.237 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2021-09-17T10:48:19.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ERROR 2021-09-17T10:48:19.518 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-53 2021-09-17T10:48:19.841 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2021-09-17T10:48:19.845 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2021-09-17T10:48:19.846 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2021-09-17T10:48:19.846 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2021-09-17T10:48:19.847 INFO:tasks.rgw.c1.client.1:Sent signal 15 2021-09-17T10:48:19.848 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2021-09-17T10:48:19.849 INFO:tasks.ceph.c1.osd.0.smithi018.stderr:2021-09-17T10:48:19.863+0000 7f524c052700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 14049) UID: 0 2021-09-17T10:48:19.850 INFO:tasks.ceph.c1.osd.0.smithi018.stderr:2021-09-17T10:48:19.863+0000 7f524c052700 -1 osd.0 60 *** Got signal Terminated *** 2021-09-17T10:48:19.850 INFO:tasks.ceph.c1.osd.0.smithi018.stderr:2021-09-17T10:48:19.863+0000 7f524c052700 -1 osd.0 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-17T10:48:19.851 INFO:tasks.ceph.c1.osd.1.smithi018.stderr:2021-09-17T10:48:19.867+0000 7f289f5de700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 14052) UID: 0 2021-09-17T10:48:19.852 INFO:tasks.ceph.c1.osd.1.smithi018.stderr:2021-09-17T10:48:19.867+0000 7f289f5de700 -1 osd.1 60 *** Got signal Terminated *** 2021-09-17T10:48:19.852 INFO:tasks.ceph.c1.osd.1.smithi018.stderr:2021-09-17T10:48:19.867+0000 7f289f5de700 -1 osd.1 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-17T10:48:19.853 INFO:tasks.ceph.c1.osd.2.smithi018.stderr:2021-09-17T10:48:19.867+0000 7fdd9f1e6700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 14053) UID: 0 2021-09-17T10:48:19.853 INFO:tasks.ceph.c1.osd.2.smithi018.stderr:2021-09-17T10:48:19.867+0000 7fdd9f1e6700 -1 osd.2 60 *** Got signal Terminated *** 2021-09-17T10:48:19.854 INFO:tasks.ceph.c1.osd.2.smithi018.stderr:2021-09-17T10:48:19.867+0000 7fdd9f1e6700 -1 osd.2 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-17T10:48:19.855 INFO:tasks.ceph.c1.mon.a.smithi018.stderr:2021-09-17T10:48:19.867+0000 7fa7d8428700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 13866) UID: 0 2021-09-17T10:48:19.855 INFO:tasks.ceph.c1.mon.a.smithi018.stderr:2021-09-17T10:48:19.867+0000 7fa7d8428700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-09-17T10:48:19.856 INFO:tasks.rgw.c1.client.1.smithi018.stdout:2021-09-17T10:48:19.871+0000 7fe5f5ef4700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 20804) UID: 0 2021-09-17T10:48:19.859 INFO:tasks.rgw.c1.client.1.smithi018.stdout:2021-09-17T10:48:19.871+0000 7fe5fa3bfb40 -1 shutting down 2021-09-17T10:49:33.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ERROR 2021-09-17T10:49:33.672 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-54 2021-09-17T10:50:13.372 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2021-09-17T10:50:13.373 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gopfzo-55 2021-09-17T10:50:52.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2021-09-17T10:50:52.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:50:52.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T10:50:52.221 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T10:50:52.262 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T10:50:52.263 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T10:55:52.281 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T10:55:52.302+0000 7ff23cf40b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:00:52.281 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T11:00:52.303+0000 7ff23cf40b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:05:52.282 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T11:05:52.304+0000 7ff23cf40b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:10:52.297 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T11:10:52.304+0000 7ff23cf40b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:15:52.306 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T11:15:52.309+0000 7ff23cf40b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:20:52.312 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T11:20:52.310+0000 7ff23cf40b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:25:52.316 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T11:25:52.310+0000 7ff23cf40b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:30:52.320 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T11:30:52.310+0000 7ff23cf40b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:35:52.327 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T11:35:52.311+0000 7ff23cf40b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:40:52.324 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T11:40:52.311+0000 7ff23cf40b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:40:52.325 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T11:40:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T11:40:52.328 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2021-09-17T11:40:52.329 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-09-17T11:40:52.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T11:40:52.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T11:40:52.336 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T11:40:52.377 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T11:40:52.377 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T11:45:52.375 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T11:45:52.364+0000 7fa9fd971b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:50:52.374 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T11:50:52.365+0000 7fa9fd971b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T11:55:52.373 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T11:55:52.366+0000 7fa9fd971b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:00:52.374 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T12:00:52.366+0000 7fa9fd971b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:05:52.375 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T12:05:52.367+0000 7fa9fd971b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:10:52.376 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T12:10:52.368+0000 7fa9fd971b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:15:52.375 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T12:15:52.368+0000 7fa9fd971b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:20:52.375 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T12:20:52.369+0000 7fa9fd971b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:25:52.375 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T12:25:52.370+0000 7fa9fd971b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:30:52.376 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T12:30:52.371+0000 7fa9fd971b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:30:52.377 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T12:30:52.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T12:30:52.381 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... FAIL 2021-09-17T12:30:52.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T12:30:52.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T12:30:52.384 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T12:30:52.466 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T12:30:52.466 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T12:35:52.470 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T12:35:52.463+0000 7f713ba0eb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:40:52.470 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T12:40:52.468+0000 7f713ba0eb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:45:52.471 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T12:45:52.468+0000 7f713ba0eb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:50:52.470 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T12:50:52.469+0000 7f713ba0eb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T12:55:52.471 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T12:55:52.470+0000 7f713ba0eb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:00:52.471 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T13:00:52.470+0000 7f713ba0eb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:05:52.472 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T13:05:52.467+0000 7f713ba0eb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:10:52.473 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T13:10:52.472+0000 7f713ba0eb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:15:52.473 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T13:15:52.472+0000 7f713ba0eb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:20:52.474 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T13:20:52.473+0000 7f713ba0eb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:20:52.499 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T13:20:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T13:20:52.503 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2021-09-17T13:20:52.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T13:20:52.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T13:20:52.506 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T13:20:52.549 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T13:20:52.549 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T13:25:52.552 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T13:25:52.549+0000 7f882335cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:30:52.552 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T13:30:52.550+0000 7f882335cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:35:52.553 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T13:35:52.550+0000 7f882335cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:40:52.554 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T13:40:52.551+0000 7f882335cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:45:52.557 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T13:45:52.551+0000 7f882335cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:50:52.556 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T13:50:52.551+0000 7f882335cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T13:55:52.557 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T13:55:52.552+0000 7f882335cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T14:00:52.558 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T14:00:52.556+0000 7f882335cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T14:05:52.559 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T14:05:52.556+0000 7f882335cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T14:10:52.560 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T14:10:52.557+0000 7f882335cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T14:10:52.561 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T14:10:52.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T14:10:52.564 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2021-09-17T14:10:52.565 INFO:tasks.rgw_multisite_tests:test s3 notification filter on master ... ok 2021-09-17T14:10:52.566 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_object_timing ... ok 2021-09-17T14:10:52.566 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2021-09-17T14:10:52.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T14:10:52.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T14:10:52.569 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T14:10:52.611 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T14:10:52.611 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T14:15:52.615 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T14:15:52.609+0000 7fdf32eeab00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T14:20:52.616 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T14:20:52.613+0000 7fdf32eeab00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T14:25:52.616 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T14:25:52.614+0000 7fdf32eeab00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T14:30:52.617 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T14:30:52.610+0000 7fdf32eeab00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T14:35:52.618 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T14:35:52.611+0000 7fdf32eeab00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T14:40:52.619 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T14:40:52.615+0000 7fdf32eeab00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T14:45:52.624 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T14:45:52.616+0000 7fdf32eeab00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T14:50:52.620 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T14:50:52.616+0000 7fdf32eeab00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T14:55:52.620 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T14:55:52.617+0000 7fdf32eeab00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T15:00:52.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T15:00:52.628 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T15:00:52.618+0000 7fdf32eeab00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T15:00:52.628 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T15:00:52.630 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... FAIL 2021-09-17T15:00:52.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T15:00:52.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T15:00:52.633 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T15:00:52.708 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T15:00:52.708 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T15:05:52.711 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T15:05:52.706+0000 7f859c7bdb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T15:10:52.712 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T15:10:52.707+0000 7f859c7bdb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T15:15:52.712 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T15:15:52.708+0000 7f859c7bdb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T15:20:52.713 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T15:20:52.708+0000 7f859c7bdb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T15:25:52.713 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T15:25:52.709+0000 7f859c7bdb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T15:30:52.713 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T15:30:52.710+0000 7f859c7bdb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T15:35:52.715 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T15:35:52.710+0000 7f859c7bdb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T15:40:52.712 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T15:40:52.711+0000 7f859c7bdb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T15:45:52.710 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T15:45:52.712+0000 7f859c7bdb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T15:50:52.709 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T15:50:52.712+0000 7f859c7bdb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T15:50:52.710 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T15:50:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T15:50:52.715 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... FAIL 2021-09-17T15:50:52.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T15:50:52.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T15:50:52.718 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T15:50:52.758 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T15:50:52.759 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T15:55:52.760 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T15:55:52.765+0000 7ff6959e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T16:00:52.761 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T16:00:52.766+0000 7ff6959e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T16:05:52.759 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T16:05:52.766+0000 7ff6959e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T16:10:52.759 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T16:10:52.767+0000 7ff6959e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T16:15:52.758 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T16:15:52.768+0000 7ff6959e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T16:20:52.758 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T16:20:52.768+0000 7ff6959e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T16:25:52.758 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T16:25:52.769+0000 7ff6959e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T16:30:52.758 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T16:30:52.770+0000 7ff6959e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T16:35:52.757 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T16:35:52.770+0000 7ff6959e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T16:40:52.757 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T16:40:52.771+0000 7ff6959e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T16:40:52.759 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T16:40:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T16:40:52.764 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... FAIL 2021-09-17T16:40:52.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T16:40:52.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T16:40:52.767 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T16:40:52.846 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T16:40:52.847 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T16:45:52.849 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T16:45:52.864+0000 7f1cfe429b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T16:50:52.857 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T16:50:52.864+0000 7f1cfe429b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T16:55:52.864 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T16:55:52.865+0000 7f1cfe429b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T17:00:52.869 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T17:00:52.866+0000 7f1cfe429b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T17:05:52.872 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T17:05:52.866+0000 7f1cfe429b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T17:10:52.874 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T17:10:52.867+0000 7f1cfe429b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T17:15:52.876 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T17:15:52.868+0000 7f1cfe429b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T17:20:52.878 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T17:20:52.868+0000 7f1cfe429b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T17:25:52.879 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T17:25:52.869+0000 7f1cfe429b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T17:30:52.877 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T17:30:52.869+0000 7f1cfe429b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T17:30:52.910 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T17:30:52.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T17:30:52.914 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... FAIL 2021-09-17T17:30:52.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T17:30:52.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T17:30:52.917 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T17:30:52.959 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T17:30:52.959 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T17:35:52.962 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T17:35:52.958+0000 7fe57617cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T17:40:52.962 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T17:40:52.958+0000 7fe57617cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T17:45:52.963 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T17:45:52.959+0000 7fe57617cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T17:50:52.964 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T17:50:52.959+0000 7fe57617cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T17:55:52.965 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T17:55:52.960+0000 7fe57617cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T18:00:52.963 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T18:00:52.961+0000 7fe57617cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T18:05:52.963 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T18:05:52.961+0000 7fe57617cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T18:10:52.963 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T18:10:52.962+0000 7fe57617cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T18:15:52.965 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T18:15:52.962+0000 7fe57617cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T18:20:52.965 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T18:20:52.963+0000 7fe57617cb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T18:20:52.966 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T18:20:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T18:20:52.971 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... FAIL 2021-09-17T18:20:52.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T18:20:52.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T18:20:52.975 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T18:20:53.018 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T18:20:53.018 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T18:25:53.021 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T18:25:53.019+0000 7f51fc073b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T18:30:53.022 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T18:30:53.020+0000 7f51fc073b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T18:35:53.023 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T18:35:53.020+0000 7f51fc073b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T18:40:53.025 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T18:40:53.021+0000 7f51fc073b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T18:45:53.026 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T18:45:53.021+0000 7f51fc073b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T18:50:53.027 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T18:50:53.022+0000 7f51fc073b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T18:55:53.030 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T18:55:53.022+0000 7f51fc073b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T19:00:53.029 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T19:00:53.023+0000 7f51fc073b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T19:05:53.030 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T19:05:53.023+0000 7f51fc073b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T19:10:53.030 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T19:10:53.024+0000 7f51fc073b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T19:10:53.031 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T19:10:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T19:10:53.040 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2021-09-17T19:10:53.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T19:10:53.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T19:10:53.043 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T19:10:53.124 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T19:10:53.124 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T19:15:53.127 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T19:15:53.120+0000 7fa45a446b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T19:20:53.129 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T19:20:53.121+0000 7fa45a446b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T19:25:53.128 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T19:25:53.121+0000 7fa45a446b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T19:30:53.129 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T19:30:53.126+0000 7fa45a446b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T19:35:53.130 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T19:35:53.126+0000 7fa45a446b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T19:40:53.131 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T19:40:53.127+0000 7fa45a446b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T19:45:53.137 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T19:45:53.128+0000 7fa45a446b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T19:50:53.132 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T19:50:53.128+0000 7fa45a446b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T19:55:53.133 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T19:55:53.129+0000 7fa45a446b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T20:00:53.134 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T20:00:53.129+0000 7fa45a446b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T20:00:53.135 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T20:00:53.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T20:00:53.146 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... FAIL 2021-09-17T20:00:53.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T20:00:53.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T20:00:53.149 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T20:00:53.194 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T20:00:53.194 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T20:05:53.197 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T20:05:53.190+0000 7fbd67a44b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T20:10:53.198 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T20:10:53.190+0000 7fbd67a44b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T20:15:53.194 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T20:15:53.191+0000 7fbd67a44b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T20:20:53.192 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T20:20:53.191+0000 7fbd67a44b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T20:25:53.190 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T20:25:53.192+0000 7fbd67a44b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T20:30:53.191 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T20:30:53.192+0000 7fbd67a44b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T20:35:53.189 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T20:35:53.193+0000 7fbd67a44b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T20:40:53.188 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T20:40:53.194+0000 7fbd67a44b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T20:45:53.188 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T20:45:53.194+0000 7fbd67a44b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T20:50:53.195 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T20:50:53.195+0000 7fbd67a44b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T20:50:53.196 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T20:50:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T20:50:53.200 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... FAIL 2021-09-17T20:50:53.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T20:50:53.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T20:50:53.203 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T20:50:53.245 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T20:50:53.245 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T20:55:53.251 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T20:55:53.252+0000 7f507c75bb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:00:53.254 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T21:00:53.252+0000 7f507c75bb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:05:53.257 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T21:05:53.253+0000 7f507c75bb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:10:53.257 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T21:10:53.254+0000 7f507c75bb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:15:53.258 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T21:15:53.254+0000 7f507c75bb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:20:53.260 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T21:20:53.255+0000 7f507c75bb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:25:53.259 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T21:25:53.256+0000 7f507c75bb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:30:53.260 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T21:30:53.256+0000 7f507c75bb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:35:53.265 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T21:35:53.257+0000 7f507c75bb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:40:53.262 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T21:40:53.258+0000 7f507c75bb00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:40:53.263 INFO:teuthology.orchestra.run.smithi018.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T21:40:53.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T21:40:53.267 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... FAIL 2021-09-17T21:40:53.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T21:40:53.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T21:40:53.270 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T21:40:53.351 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setuser ceph since I am not root 2021-09-17T21:40:53.352 INFO:teuthology.orchestra.run.smithi018.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T21:45:53.354 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T21:45:53.350+0000 7f27941e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:50:53.355 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T21:50:53.351+0000 7f27941e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:55:53.356 INFO:teuthology.orchestra.run.smithi018.stderr:2021-09-17T21:55:53.352+0000 7f27941e4b00 0 monclient(hunting): authenticate timed out after 300 2021-09-17T21:57:12.978 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-09-17T21:57:13.012 DEBUG:teuthology.task.console_log:Killing console logger for smithi018 2021-09-17T21:57:13.013 DEBUG:teuthology.task.console_log:Killing console logger for smithi050 2021-09-17T21:57:13.014 DEBUG:teuthology.exit:Finished running handlers